...

The following illustration is a typical, although simplified, Universal Data Manager Gateway system in a High Availability environment.

...

The mode (status) of a cluster node indicates whether or not it is the cluster node that currently is processing work:

Active

Cluster node currently is performing all file transfers and the also the other system processing functions (administration, reporting, …).

It supports the local services and the processing of transfers in server and client modes.

Passive

Cluster Node is not performing any file transfers but is available to perform the other system processing functions.

Offline

Cluster node is not running or is inoperable and needs to be restarted.

High Availability Start-Up

The following steps describe how a High Availability environment starts up:

Step 1	User starts the Cluster Nodes.
Step 2	Each cluster node reads its `server.ini` file.
Step 3	Each cluster node locates and connects to the database and retrieves information about the UDMG environment.
Step 4	One of the nodes becomes the Active node. It starts the local services (local FTP, SFTP, ... servers) and starts the processing of transfer requests, both in server mode and client mode.

Note

title	Note

Cluster nodes in Passive mode can perform limited system processing functions.

Determining Mode of a Cluster Node at Start-up

A cluster node starts in Passive mode. It then determines if it should remain in Passive mode or switch to Active mode.

The following flow chart describes how a cluster node determines its mode at start-up:

Image Modified

Note

title	Note

A cluster node is considered "healthy" or "stale" based on its heartbeat timestamp.

Checking the Active Cluster Node During Operations

When all cluster nodes have started, each one continuously monitors the heartbeats of the other running cluster nodes.

If a Passive cluster node determines that the Active cluster node is no longer running, the Passive cluster node automatically takes over as the Active cluster node based upon the same criteria described above.

This determination is made as follows:

Step 1	The Active cluster node sends a heartbeat by updating a timestamp in the database. The heartbeat interval is 10

...

seconds

...

. It can be adjusted with the 'Heartbeat' parameter:

[controller] ; The frequency at which the heartbeat will be updated Heartbeat = 10s

Step 2

All Passive cluster nodes check the Active cluster node's timestamp to determine if it is current.

...

This check runs every

...

20 seconds. It can be adjusted with the 'HeartbeatCheck' parameter:

[controller] ; The heartbeat to determine if this instance will be probed HeartbeatCheck = 20s

Step 3

If a Passive cluster node determines that the Active cluster node's timestamp is stale, failover occurs: the Passive cluster node changes the mode of the Active cluster node to Offline and takes over as the Active cluster node. If more than one cluster node is operating in Passive mode, the first cluster node eligible to become Active that determines that the Active cluster node is not running becomes the Active cluster node. A stale cluster node is one whose timestamp is older than 5 minutes.

It can be adjusted with the 'Deadline' parameter:

[controller] ; The deadline to determine if this instance will be active Deadline = 5m

What To Do If a Failover Occurs

A Passive cluster node taking over as an Active cluster node is referred to as failover. If failover occurs, the event is invisible unless you are using the Active cluster node in a browser.

If you are using the Active cluster node in a browser and the cluster node fails, you will receive a browser error. In this case, take the following steps to continue working:

Step 1

Access the new Active cluster node in your browser.

This can be achieved by selecting the appropriate service in that UDMG Admin UI login page, provided that each UDMG Server instance is defined as a dedicated login service.

Step 2

If you were adding, deleting, or updating records at the time of the failure, check the record you were working on. Any data you had not yet saved will be lost.

Viewing Cluster Node Status

The cluster node status is displayed by the background color of the Server Status button.

...

It is also indicated by the "Controller" service Information string:

High Availability Configuration

To achieve High Availability for your Universal Data Mover Gateway, you must configure the UDMG Server cluster nodes, UDMG Authentication Proxy.

Configuring Cluster Nodes

All cluster nodes in a High Availability environment must point to the same database by making sure the following entries in their server.ini files are the same.

For example:

Code Block

[database]
; Type of the RDBMS used for the gateway database. Possible values: sqlite, mysql (default), postgresql, oracle, mssql
Type = postgresql

; Address (URL:port) of the database. The default port depends on the type of database used (PostgreSQL: 5432, MySQL: 3306, MS SQL: 1433, Oracle: 1521, SQLite: none).
Address = localhost:5432

; The name of the database
Name = udmg

; The name of the gateway database user
User = udmg_user

; The password of the gateway database user
Password = udmg_password

; Path of the database TLS certificate file (only supported for mysql, postgresql).
; TLSCert =

; Path of the key of the TLS certificate file (only supported for mysql, postgresql).
; TLSKey =

; The path to the file containing the passphrase used to encrypt account passwords using AES. Recommended to be a full absolute path, if the file does not exist, a new passphrase is generated the first time.
AESPassphrase = /opt/udmg/etc/udmg-server/passphrase.aes

The heartbeat frequency and failover deadline can also be tuned with the following parameters:

Code Block

[controller]
; The frequency at which the heartbeat will be updated
Heartbeat = 10s

; The deadline to determine if this instance will be active
Deadline = 5m

; The heartbeat to determine if this instance will be probed
HeartbeatCheck = 20s

Load Balancer

REST API connections

If you are using a load balancer in your High Availability environment, it can utilize the following HTTP requests to direct the UDMG Admin UI and REST API requests to the active instance:

http(s)://serverhost:[Port]/ping

If a cluster node is active, this URL returns the status

...

200 (OK) and a simple one word content

...

of ACTIVE.

If a cluster node is not active, this URL returns the status

...

400 (cluster node is not active) and lists the actual mode of the cluster node:

...

PASSIVE or

...

OFFLINE.

This API is provided without authentication.

http(s)://serverhost:[Port]/api/sb_healthcheck

This URL returns information about a cluster node:

Code Block

{
    "status": "operational",
    "nodeId": "gateway_1:8080-mft-gw-0",
    "nodeHostname": "gateway_1",
    "nodeIPAddress": "172.99.0.101",
    "nodePort": "8080",
    "nodeStatus": "PASSIVE",
    "nodeUptime": "50h1m56.09356413s",
    "nodeLastUpdate": "2023-11-15T19:03:30.481154Z",
    "nodeLastActiveDate": "2023-11-13T16:57:57.026091Z"
}

This API requires authentication but no specific permissions.

http(s)://serverhost:[Port]/api/sb_mgmt_nodes

This URL returns information about all the cluster nodes:

Code Block

{
    "nodes": [
        {
            "nodeId": "gateway:8080-mft-gw-0",
            "nodeHostname": "gateway",
            "nodeIPAddress": "172.99.0.100",
            "nodePort": "8080",
            "nodeStatus": "ACTIVE",
            "nodeUptime": "16m15.413255244s",
            "nodeLastUpdate": "2023-11-09T15:24:20.562225Z",
            "nodeLastActiveDate": "2023-11-09T15:08:40.105002Z" 
        },
        {
            "nodeId": "gateway_1:8080-mft-gw-0",
            "nodeHostname": "gateway_1",
            "nodeIPAddress": "172.99.0.101",
            "nodePort": "8080",
            "nodeStatus": "PASSIVE",
            "nodeUptime": "16m0.631810433s",
            "nodeLastUpdate": "2023-11-09T15:24:35.289412Z",
            "nodeLastActiveDate": "2023-11-09T14:40:28.491424Z" 
        }
    ]
}

This API requires authentication and the 'administration read' permission.

This can be used to direct the UDMG Admin UI and REST API requests to the active instance.

Local MFT server connections

Until the node is marked as 'ACTIVE', the local server services that controlled by UDMG Server are not be started and there is no listener on their defined port.

As there can be some delay between the node status change and the actual start and stop of each local MFT server, it is recommended to use a TCP connection test to determine which node is currently owning the listener.

Image Added

If you have the possibility of performing a double health check, you can choose to:

Validate the node status endpoint
Validate that the port is available

Example of a configuration with HAProxy:

Code Block

# --------------------------------------------------------------------------- #
# Configuration for: <service-name> 
# --------------------------------------------------------------------------- #

frontend frontend-<service-name>
  bind *:<service-port> transparent
  mode tcp
  log             global
  option tcplog
  use_backend backend-<service-name>

backend backend-<service-name>
  mode tcp
  stick-table type ip size 10k expire 300s
  stick on src
	
  # Healtcheck
  option httpchk
  http-check send meth GET uri /ping
  http-check expect status 200
  
  # Pool Servers
  server node-0 <hostname-node-0> check port <node-0-server-api-port> cookie S01  check inter 300s
  server node-1 <hostname-node-1> check port <node-1-server-api-port> cookie S02  check inter 300s

Versions Compared

Old Version 1

New Version 2

Key

High Availability Start-Up

Determining Mode of a Cluster Node at Start-up

Checking the Active Cluster Node During Operations

What To Do If a Failover Occurs

Viewing Cluster Node Status

High Availability Configuration

Configuring Cluster Nodes

Load Balancer

REST API connections

Local MFT server connections

Page Comparison

Versions Compared

Old Version 1

New Version 2

Key

High Availability Start-Up

Determining Mode of a Cluster Node at Start-up

Checking the Active Cluster Node During Operations

What To Do If a Failover Occurs

Viewing Cluster Node Status

High Availability Configuration

Configuring Cluster Nodes

Load Balancer

REST API connections

Local MFT server connections