Introduction

High Availability (HA) of Universal Data Mover Gateway means that it has been set up to be a redundant system; in addition to the components that are processing work, there are backup components available to continue processing through hardware or software failure.

This page describes a High Availability environment, with an Active-Passive setup.

High Availability System: Active-Passive

The following illustration is a typical, although simplified, Universal Data Mover Gateway Linux system in a High Availability environment.

In this environment, there are:

Two UDMG Server instances (MFT nodes)
One Shared Storage server machine. Resilience can be achieved with a cluster file system solution.
One Database Server machine. Resilience can be achieved with a cluster database solution.
Two ranges of reserved ports for FTP (3000-3010) and SFTP (4000-4010). For a given FTP or SFTP server, the same port is used on internet facing end (HAProxy) and the backend side (UDMG Server).

The components in white are active and operating. The components in gray are available for operations but currently are inactive (passive).

The Linux HAProxy and Keepalived utilities are installed to handle a virtual IP that is used by the client to reach the UDMG services.

Reference table for the sample server configuration:

IP	Hostname	Description
192.168.56.110	storage	NFS Server
192.168.56.100	vip	Virtual IP Addr
192.168.56.101	mft_1	UDMG Server Node
192.168.56.102	mft_2	UDMG Server Node
192.168.56.120	db	PostgreSQL Database

High Availability Configuration

To achieve High Availability for your Universal Data Mover Gateway system, you must configure the nodes and applications.

Configuring the NFS Server

Install the package required for the NFS server and related utilities

yum install nfs-utils

The package name could be different according to the distribution that you are using.

Add the directory that you want to share with the UDMG Servers, for example /data:

# /etc/exports
#
# See exports(5) for a description.

# use exportfs -arv to reread
/data mft_1(rw,no_subtree_check,no_root_squash)
/data mft_2(rw,no_subtree_check,no_root_squash)

Edit your /etc/hosts with the names from the Reference Table:

#/etc/hosts
127.0.0.1 localhost.localdomain localhost storage
::1 localhost localhost.local domain

192.168.56.110 storage
192.168.56.100 VIP
192.168.56.101 mft_1
192.168.56.102 mft_2

Start the NFS Service

service start NFS-server

Show all exports filesystem:

# showmount -e
Export list for storage:
/data mft_1,mft_2

Configuring the first UDMG Server

System Configuration

Configure the /etc/hosts file with the following entries:

#/etc/hosts
192.168.56.110 storage
192.168.56.100 vip
192.168.56.101 mft_1
192.168.56.102 mft_2
192.168.56.120 db

Enable this option in the /etc/sysctl.conf, this allows HAProxy to bind to the shared IP address, that will be defined with Keepalived:

net.ipv4.ip_nonlocal_bind = 1

Install the application packages for HAProxy, Keepalived, and the NFS client utilities:

mft:~# yum install haproxy keepalived nfs-utils

Configuring NFS Client

Check if you can mount the filesystem from the storage server

mft:~# showmount -e storage
Export list for storage:
/data mft_1,mft_2

Edit /etc/fstab to mount the remote filesystem under the server:

storage:/data /data nfs defaults,nolock,vers=3,soft 0 0

Using NFSv3 as NFSv4 does not support the nolock option, without this option the server could hang.

Mount all the filesystem:

mount -a

Verify that the filesystem is mounted and writable:

mft:~# df -h /data
Filesystem Size Used Available Use% Mounted on
storage:/data 15.6G 173.0M 14.6G 1% /data

mft:~# touch /data/test.1.txt
mft:~# stat /data/test.1.txt
File: /data/test.1.txt
Size: 0 Blocks: 0 IO Block: 262144 regular empty file
Device: 1bh/27d Inode: 260790 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2022-09-28 16:38:01.489853504 +0000
Modify: 2022-09-28 16:38:01.489853504 +0000
Change: 2022-09-28 16:38:01.489853504 +0000

Validate that the file exists on the storage server before continuing.

Configuring Keepalived

Now we are going to configure the keepalived service.

Create the following configuration under /etc/keepalived/keepalived.conf

vrrp_instance VI_1 {
state MASTER # This setting must be comment for Backup Mode
# state BACKUP # This setting will be uncommented for the Backup Node
interface eth1
virtual_router_id 51
priority 255
advert_int 1
authentication {
auth_type PASS
auth_pass 12345
}
virtual_ipaddress {
192.168.56.100/24 dev eth1 label eth1:1
}
}

In our case, the VIP interface will be under the interface eth1

Before starting the service, check the list of network interfaces:

# ifconfig -a
eth0 Link encap:Ethernet HWaddr 08:00:27:1E:96:CF
inet addr:10.0.2.15 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe1e:96cf/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:19 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1180 (1.1 KiB) TX bytes:1970 (1.9 KiB)

eth1 Link encap:Ethernet HWaddr 08:00:27:E3:61:8C
inet addr:192.168.56.101 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fee3:618c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6322 errors:0 dropped:0 overruns:0 frame:0
TX packets:19415 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1181451 (1.1 MiB) TX bytes:1548756 (1.4 MiB)

lo. Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:96944 errors:0 dropped:0 overruns:0 frame:0
TX packets:96944 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:5988148 (5.7 MiB) TX bytes:5988148 (5.7 MiB)

Start the service

# service keepalived start

Now you should see a network alias configured as eth1:1

mft:~# ifconfig -a
eth1 Link encap:Ethernet HWaddr 08:00:27:E3:61:8C
inet addr:192.168.56.101 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fee3:618c/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:6769 errors:0 dropped:0 overruns:0 frame:0
TX packets:20079 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1232486 (1.1 MiB) TX bytes:1609240 (1.5 MiB)

eth1:1 Link encap:Ethernet HWaddr 08:00:27:E3:61:8C
inet addr:192.168.56.100 Bcast:0.0.0.0 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

This is the virtual IP configured by Keepalived

Configuring HAProxy

Create the following configuration file /etc/haproxy/haproxy.cfg

The placeholder "<SERVERNAME>" must be the server name of the configuration. Since we are going to install this under 2 machines, replace with mft_1 or mft_2 or localhost.
The placeholder "<SERVERPORT>" is the port of the UDMG server, 18080 with the default installation guideline.
HAProxy Status API is configured on port 8081
Two ranges of ports are forwarded for the FTP (3000-3010) and SFTP (4000-4010) inbound connections to the backend UDMG servers. This can be tuned according to the desired MFT configuration.

#/etc/haproxy/haproxy.cfg

# --------------------------------------------------------------------------- #
# Global
# --------------------------------------------------------------------------- #
global
log 127.0.0.1 local0 info

# --------------------------------------------------------------------------- #
# Defaults Timeouts
# --------------------------------------------------------------------------- #
defaults
retries 3
option redispatch
timeout client 30s
timeout connect 4s
timeout server 30s

# --------------------------------------------------------------------------- #
# Stats
# --------------------------------------------------------------------------- #
listen stats
bind *:8081
mode http
log global
maxconn 10
stats enable
stats hide-version
stats refresh 30s
stats show-node
stats auth admin:password
stats uri /status

# --------------------------------------------------------------------------- #
# FTP - mft Servers
# --------------------------------------------------------------------------- #
frontend ftp_service_front
bind vip:3000-3010 transparent
mode tcp
use_backend ftp_service_backend

backend ftp_service_backend
mode tcp
stick-table type ip size 10k expire 300s
stick on src
server gw0 <SERVERNAME> check port <SERVERPORT>

# --------------------------------------------------------------------------- #
# SFTP - mft Servers
# --------------------------------------------------------------------------- #
frontend sftp_service_front
bind vip:4000-4010 transparent
mode tcp
use_backend sftp_service_backend

backend sftp_service_backend
mode tcp
stick-table type ip size 10k expire 300s
stick on src
server gw0 <SERVERNAME> check port <SERVERPORT>

# --------------------------------------------------------------------------- #
# UDMG Server
# --------------------------------------------------------------------------- #
frontend gw_service_front
bind vip:18080 transparent
mode http
default_backend gw_service_backend

backend gw_service_backend
mode http
balance roundrobin
cookie SRVNAME insert
server gw0 <SERVERNAME> check port <SERVERPORT> cookie S01 check

# --------------------------------------------------------------------------- #
# Nginx
# --------------------------------------------------------------------------- #
frontend nx_service_front
bind vip:80 transparent
mode http
default_backend nx_service_backend

backend nx_service_backend
mode http
balance roundrobin
cookie SRVNAME insert
server gw0 <SERVERNAME> check port 80 cookie S01 check

# --------------------------------------------------------------------------- #
# END
# --------------------------------------------------------------------------- #
# EOF

Start the service:

# service haproxy start

Check if the process is running without any issue:

mft:~# ps ax | grep -i haproxy
2122 root 0:04 /usr/sbin/haproxy -D -p /var/run/haproxy.pid -f /etc/haproxy/haproxy.cfg

Verify that HAProxy is binding the ports that we are going to use under the UDMG Server:

mft:~# netstat -tanlp | grep -i haproxy
tcp 0 0 0.0.0.0:8081 0.0.0.0:* LISTEN 2122/haproxy
tcp 0 0 192.168.56.100:3000 0.0.0.0:* LISTEN 2122/haproxy
tcp 0 0 192.168.56.100:3001 0.0.0.0:* LISTEN 2122/haproxy
tcp 0 0 192.168.56.100:3002 0.0.0.0:* LISTEN 2122/haproxy
tcp 0 0 192.168.56.100:3003 0.0.0.0:* LISTEN 2122/haproxy
tcp 0 0 192.168.56.100:3004 0.0.0.0:* LISTEN 2122/haproxy
tcp 0 0 192.168.56.100:3005 0.0.0.0:* LISTEN 2122/haproxy
tcp 0 0 192.168.56.100:3006 0.0.0.0:* LISTEN 2122/haproxy
tcp 0 0 192.168.56.100:3007 0.0.0.0:* LISTEN 2122/haproxy
tcp 0 0 192.168.56.100:3008 0.0.0.0:* LISTEN 2122/haproxy
tcp 0 0 192.168.56.100:3009 0.0.0.0:* LISTEN 2122/haproxy
tcp 0 0 192.168.56.100:3010 0.0.0.0:* LISTEN 2122/haproxy
tcp 0 0 192.168.56.100:4000 0.0.0.0:* LISTEN 2122/haproxy
tcp 0 0 192.168.56.100:4001 0.0.0.0:* LISTEN 2122/haproxy
tcp 0 0 192.168.56.100:4002 0.0.0.0:* LISTEN 2122/haproxy
tcp 0 0 192.168.56.100:4003 0.0.0.0:* LISTEN 2122/haproxy
tcp 0 0 192.168.56.100:4004 0.0.0.0:* LISTEN 2122/haproxy
tcp 0 0 192.168.56.100:4005 0.0.0.0:* LISTEN 2122/haproxy
tcp 0 0 192.168.56.100:4006 0.0.0.0:* LISTEN 2122/haproxy
tcp 0 0 192.168.56.100:4007 0.0.0.0:* LISTEN 2122/haproxy
tcp 0 0 192.168.56.100:4008 0.0.0.0:* LISTEN 2122/haproxy
tcp 0 0 192.168.56.100:4009 0.0.0.0:* LISTEN 2122/haproxy
tcp 0 0 192.168.56.100:4010 0.0.0.0:* LISTEN 2122/haproxy
tcp 0 0 192.168.56.100:80 0.0.0.0:* LISTEN 2122/haproxy
tcp 0 0 192.168.56.100:18080 0.0.0.0:* LISTEN 2122/haproxy

Configuring the UDMG component

Follow the UDMG for Linux Installation guide.

UDMG Server

Under the server.ini configuration the following settings must be updated:

[global]
; The name given to identify this gateway instance. If the the database is shared between multiple gateways, this name MUST be unique across these gateways.
GatewayName = mft

[paths]
; The root directory of the gateway. By default, it is the working directory of the process.
GatewayHome = /data/root

[admin]
; The address used by the admin interface.
Host = <SERVERNAME or SERVER IP>

[database]
; The path to the file containing the passphrase used to encrypt account passwords using AES
AESPassphrase = /data/passphrase.aes

The paths and database section must be point to the /data (NFS Storage).

UDMG Authentication Proxy

Under the configuration of the proxy server.toml

# Proxy Configuration
[proxy]
recover = true
cors = true
tracker = true
logger = true
port = "5000"
inet = "127.0.0.1"

[service.mft]
protocol = "http"
policy = "failover"
admins = ["admin"]

[[service.mft.targets]]
hostname = "<SERVERNAME or SERVER IP>"
port = 18080

Start the UDMG Server and UDMG Authentication Proxy.

NGINX for UDMG Admin UI

Configure the NGINX service to reach the IP that was configured before:

upstream udmg_auth_proxy {
ip_hash;
server <SERVERNAME or SERVER IP>:5000;
keepalive 10;
}

server {
listen <SERVERNAME or SERVER IP>::80 default_server;

location / {
try_files $uri $uri/ /index.html;
root "/var/www/localhost/htdocs";
}

location /service/ {
proxy_pass http://udmg_auth_proxy/;
}

# You may need this to prevent return 404 recursion.
location = /404.html {
internal;
}
}

Start the nginx service.

Configuring the second UDMG Server

Repeat the above steps with the following difference regarding the state of the virtual IP.

Keepalived

Create the following configuration under /etc/keepalived/keepalived.conf

vrrp_instance VI_1 {
# state MASTER # This setting must be comment for Backup Mode
state BACKUP # This setting will be uncommented for the Backup Node
interface eth1
virtual_router_id 51
priority 255
advert_int 1
authentication {
auth_type PASS
auth_pass 12345
}
virtual_ipaddress {
192.168.56.100/24 dev eth1 label eth1:1
}
}

Checking the failover

In order to see whether the configuration was successful, stop one of the the mft server, and validate that the VIP interface is moving to the other host.

References

This document references the following documents.

Name	Location
Setting up a Linux cluster with Keepalived	https://www.redhat.com/sysadmin/keepalived-basics

Universal Data Mover Gateway 1.3

High Availability: Linux Active-Passive

Introduction

High Availability System: Active-Passive

High Availability Configuration

Configuring the NFS Server

Configuring the first UDMG Server

System Configuration

Configuring NFS Client

Configuring Keepalived

Configuring HAProxy

Configuring the UDMG component

UDMG Server

UDMG Authentication Proxy

NGINX for UDMG Admin UI

Configuring the second UDMG Server

Keepalived

Checking the failover

References