Pre-Installation / Upgrade Backups

The installation process overwrites the current files (exception: the configuration files are kept), effectively removing your modifications. Backing up these files will optimize the time it takes you to get up and running after installing or upgrading.

After upgrading RPM or DEB packages, review the new configuration file templates (with the extension .rpmnew or .dpk-new) and edit the current configuration files to add new parameters or remove deprecated parameters.

Note

If you are upgrading an installation of UDMG from any release prior to 1.3.0.0, you must uninstall the older version before installing the new version.
The installation packages, binaries, services, and environment variables have changed and this does not allow for a standard upgrade.

Category	Prior releases	Release 1.3
user and group	mft:mft	udmg:udmg
binaries	/usr/local/bin	/opt/udmg/bin
configuration files	/etc/mft	/opt/udmg/etc
log files	/var/opt/udmg/logs	/var/opt/udmg/logs
UDMG Admin UI assets	/opt/udmg/var/www/mft	/opt/udmg/var/www/udmg
Services	mft_waarp_gateway mft_auth_proxy nginx mft_web_transfer_client mft-agent-proxy-client mft-agent-proxy-server	udmg-server udmg-auth-proxy nginx udmg-web-transfer udmg-agent-server udmg-agent-client
Environment variables	WAARP_GATEWAY_ADDRESS MFT_AUTH_PROXY_CONFIG MFT_AGENT_PROXY_CONFIG	UDMG_SERVER_ADDRESS UDMG_AUTH_PROXY_CONFIG UDMG_AGENT_PROXY_CONFIG

It may be required to modify the work and data directories ownership or access rights and to update UDMG Server transfer rules to use paths that are accessible by the 'udmg' user.

The configuration files must be reviewed and compared between the old and new locations.

Special attention is required for the AESpassphrase parameter for UDMG Server. It must be the path for the file that was used by the previous release and must be accessible by the new service user. It is recommened to set an absolute path in the configuration file.

Upgrading UDMG Server

The UDMG release version is stored in the database to ensure the data structure is compatbile with the version of the UDMG components.

After upgrading the component binaries and before starting the UDMG server it is required to perform the release migration step.

The udmg-server "migrate" command handles the necessay database updates and the setting of the internal version.

$ /opt/udmg/bin/udmg-server migrate --help
Usage:
  udmg-server [OPTIONS] migrate [migrate-OPTIONS] [version]

Help Options:
  -h, --help         Show this help message

[migrate command options]
      -c, --config=  The configuration file to use
      -d, --dry-run  Simulate the migration but does not commit the changes
      -l, --list     List Migrations
      -f, --file=    Writes the migration commands into a file instead of sending them to the database
      -v, --verbose  Show verbose debug information. Can be repeated to increase verbosity

[migrate command arguments]
  version:           The version to which the database should be migrated

The configuration file is the one that will be used for the server mode, with the parameter for accessing the target database.

To get the list of supported target version, use the list parameter. The last version is the release version of the udmg-server.

$ /opt/udmg/bin/udmg-server migrate -c /opt/udmg/etc/udmg-server/server.ini --list | tail -n 5
0.7.1
0.7.1-sb.1
0.7.1-sb.2
0.7.1-sb.3
1.3.0

It is recommended to set the verbose parameter (3 times) to follow the progress.

$ /opt/udmg/bin/udmg-server migrate -c /opt/udmg/etc/udmg-server/server.ini -vvv 1.3.0.0
[INFO ] Migration: Starting upgrade migration...
[INFO ] Migration: Applying migration 'Fix the 'sb_generic_group_join' table foreign keys'
[INFO ] Migration: Applying migration 'Fix the 'sb_user_group_member' table foreign keys'
[INFO ] Migration: Applying migration 'Fix the 'sb_user_group_permission' table foreign keys'
[INFO ] Migration: Applying migration 'Bump database version to 0.7.1-sb.2'
[INFO ] Migration: Applying migration 'Add description under 'local_agents' and 'remote_accounts''
[INFO ] Migration: Applying migration 'Add a 'revoked' column to the 'crypto_credentials' table'
[INFO ] Migration: Applying migration 'Bump database version to 0.7.1-sb.3'
[INFO ] Migration: Applying migration 'Add a 'email' column to the 'users' table'
[INFO ] Migration: Applying migration 'Create the 'access_token' table'
[INFO ] Migration: Applying migration 'Bump database version to 1.3.0'

After the migration, the service for UDMG Server can be started.

Installing and Configuring the Components

UDMG Admin UI

The following steps require root privilege, make sure that you have the correct access before continuing.

Extract the distribution file for UDMG Admin UI, under the directory web server root directory, see the NGINX Service configuration above.

# unzip -d /opt/udmg/var/www/udmg/ udmg_admin_ui-<VERSION>.zip

Validate that the service is working properly:

# curl http://localhost:80 -I
HTTP/1.1 200 OK
Server: nginx/1.21.6
Date: Mon, 06 Jun 2022 17:33:19 GMT
Content-Type: text/html
Content-Length: 7788
Last-Modified: Fri, 03 Jun 2022 14:07:05 GMT
Connection: keep-alive
ETag: "629a1589-1e6c"
Accept-Ranges: bytes

UDMG User setup

Create a dedicated user for running the UDMG modules and to be the owner of the files that will be transferred by UDMG.

# groupadd udmg

# useradd -g udmg udmg

UDMG Server

Create the configuration file /opt/udmg/etc/udmg-server/server.ini with the following parameters:

# mkdir -p /opt/udmg/etc/udmg-server
# vi /opt/udmg/etc/udmg-server/server.ini

Note

The lines starting with a semicolon ';' or a hash '#' are comments, describing the option or showing the default value.

The parameters must be adapted to your environment, in particular:

global: GatewayHome
log: LogLevel, LogTo, LogPath
admin: Host, Port
database: Type, Address, Name, User, Password

[global]
; The name given to identify this gateway instance. If the the database is shared between multiple gateways, this name MUST be unique across these gateways.
GatewayName = udmg

; Default OS permission for created files
; FilePermissions = 700

; Default OS permission for created directories
; DirPermissions = 750

[paths]
; The root directory of the gateway. By default, it is the working directory of the process.
GatewayHome = /home/udmg

; The directory for all incoming files.
; DefaultInDir = in

; The directory for all outgoing files.
; DefaultOutDir = out

; The directory for all running transfer files.
; DefaultTmpDir = tmp

[log]
; All messages with a severity above this level will be logged. Possible values are DEBUG, INFO, WARNING, ERROR and CRITICAL.
Level = DEBUG

; The path to the file where the logs must be written. Special values 'stdout' and 'syslog' log respectively to the standard output and to the syslog daemon
; LogTo = stdout

; If LogTo is set on 'syslog', the logs will be written to this facility.
; SyslogFacility = local0

; The directory for the log files of the local servers, partners, and transfers. No default, if not provided then the detailled log feature is disabled.
; LogPath = /var/opt/udmg/logs

[admin]
; The address used by the admin interface.
Host = 0.0.0.0

; The port used by the admin interface. If the port is 0, a free port will automatically be chosen.
Port = 18080

; Path of the TLS certificate for the admin interface.
; TLSCert =

; Path of the key of the TLS certificate.
; TLSKey =

; API rate limiter: number of allowed requests per client IP, per second. After that HTTP code 429 is returned. Disabled if 0 or not provided.

; RateLimit = 0

[database]
; Name of the RDBMS used for the gateway database. Possible values: sqlite, mysql, postgresql
Type = postgresql

; Address of the database
Address = localhost

; The name of the database
Name = udmg

; The name of the gateway database user
User = udmg_user

; The password of the gateway database user
Password = udmg_password

; Path of the database TLS certificate file.
; TLSCert =

; Path of the key of the TLS certificate file.
; TLSKey =

; The path to the file containing the passphrase used to encrypt account passwords using AES
; AESPassphrase = passphrase.aes

[controller]
; The frequency at which the database will be probed for new transfers
; Delay = 5s

; The maximum number of concurrent incoming transfers allowed on the gateway (0 = unlimited).
; MaxTransferIn = 0

; The maximum number of concurrent outgoing transfers allowed on the gateway (0 = unlimited).
; MaxTransferOut = 0

[sftp]
; Set to true to allow legacy and weak cipher algorithms: 3des-cbc, aes128-cbc, arcfour, arcfour128, arcfour256
; AllowLegacyCiphers = false

Note

AESPassphrase file is generated on first run if it does not exist. It is recommended to set an absolute path, otherwise it is created in the curent directory.

Make sure to set verify the file location during an upgrade and to have a backup. Without the correct AESPassphrase file, the passwords, the keys and the certificates will not be usable.

Install the binaries under /opt/udmg/bin:

# install -m 755 udmg-client /opt/udmg/bin
# install -m 755 udmg-server /opt/udmg/bin

UDMG Authentication Proxy

Create a directory under /etc/udmg/:

# mkdir -p /opt/udmg/etc/udmg

Create a configuration file for the service:

# vi /opt/udmg/etc/udmg/auth_proxy/config.toml

# Proxy Configuration
[proxy]
# Port, default "5000"
port = "5000"
# Network interface, default "0.0.0.0"
inet = "127.0.0.1"
# Enable recover on panic, default true, should be true for production environment
recover = true
# Enable Cross-Origin Resource Sharing (CORS), should be true for production environment
cors = true
# CORS: List of origins that may access the resource. Optional. Default value "*" # domain = "*" # Enable Request Track ID, default true
tracker = true
# Enable Request Logger, default true
logger = true
# Rate Limit IP Request over 1 second, default 0 (unlimited)
limit = 0
# Enable the Prometheus Metric Endpoint '/metric', default false
metrics = false # Enable CSRF tokencsrf = false # # Service 'local' with direct authentication on the UDMG Server
[service.local]
# UDMG Server Listen Protocol
protocol = "http"[[service.local.targets]]
# UDMG Server Hostname or IP
hostname = "localhost"
# UDMG Server Port
port = 18080

Please refer to Authentication Methods for the LDAP and SSO authentication options.

Install the binary under /opt/udmg/bin:

# install -m 755 udmg-auth-proxy /opt/udmg/bin

UDMG Agent Proxy

Create a directory under /opt/udmg/etc/udmg:

# mkdir -p /opt/udmg/etc/udmg/agent/

Install the binaries under /opt/udmg/bin:

# install -m 755 udmg-agent-proxy-client /opt/udmg/bin
# install -m 755 udmg-agent-proxy-server /opt/udmg/bin

Agent Proxy Server Configuration

Generate a SSH Key for the service:

# ssh-keygen -t rsa -q -N "" -f /opt/udmg/etc/udmg/agent/agent

Change the agent key permissions:

# chmod 755 /opt/udmg/etc/udmg/agent/agent /opt/udmg/etc/udmg/agent/agent.pub

Create a configuration file as /opt/udmg/etc/udmg/agent/agent.toml:

# vi /opt/udmg/etc/udmg/agent/agent.toml

[agent]
# UDMG Agent Proxy Hostname or IP, and port
hostname = "0.0.0.0"
port = "2222"
# path to the SSH private key file
ssh_key = "/opt/udmg/etc/udmg/agent/agent"
# path to the SSH public key file
ssh_key_pub = "/opt/udmg/etc/udmg/agent/agent.pub"

# Agent Service User and password
username = "mft"
password = "61ee8b5601a84d5154387578466c8998848ba089"

The password key will be used for the client authentication.

Agent Proxy Client Configuration

Create a configuration file as /etc/udmg/agent_proxy/client.toml:

# vi /opt/udmg/etc/udmg/agent/client.toml

[client]
# Target UDMG Agent Proxy Hostname or IP, and port
hostname = "localhost"
port = "2222"

# UDMG Agent Service User and password
username = "mft"
password = "61ee8b5601a84d5154387578466c8998848ba089"

# Default TTL to Connection Retry
ttl="5s"

[client.api]
# UDMG Agent Client Admin APIport="2280"

[gateway]
# UDMG Server Hostname or IP, and port
hostname = "localhost"
port = "18080"
# UDMG Server Username/Password
username = "admin"
password = "admin_password"

The password key will be used for the client authentication.

Setup the Systemd Services

UDMG Server

Create a new service definition:

# vi /etc/systemd/system/udmg-server.service

[Unit]
Description=UDMG Server

[Service]
Type=simple
User=udmg
Group=udmg
WorkingDirectory=/home/udmg
ExecStart=/bin/sh -c '/opt/udmg/bin/udmg-server server -c /opt/udmg/etc/udmg-server/server.ini' 
Restart=on-failure 

[Install] WantedBy=multi-user.target

Enable the new service:

# systemctl enable udmg-server.service
Created symlink /etc/systemd/system/multi-user.target.wants/udmg-server.service → /etc/systemd/system/udmg-server.service.

Start the service and check the status:

# systemctl start udmg-server
# systemctl status udmg-server
● udmg-server.service - UDMG server
Loaded: loaded ( /etc/systemd/system/udmg-server.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2022-06-07 16:43:16 -03; 10s ago
Main PID: 24888 (udmg-server)
Tasks: 6 (limit: 3509)
CPU: 11ms
CGroup: /system.slice/udmg-server.service
└─24888 /opt/udmg/bin/udmg-server server -c /opt/udmg/etc/udmg-server/server.ini

Make sure that the listen port and network interface is reachable by UDMG Authentication Proxy and UDMG Agent Client.

UDMG Authentication Proxy

Create a new service definition:

# vi /etc/systemd/system/udmg-auth-proxy.service

[Unit]
Description=UDMG Auth Proxy server

[Service]
Type=simple
User=udmg
Group=udmg
WorkingDirectory=/home/udmg
Environment="UDMG_AUTH_PROXY_CONFIG=/opt/udmg/etc/udmg/auth_proxy/config.toml"
ExecStart=/bin/sh -c 'exec /opt/udmg/bin/udmg-auth-proxy'
Restart=on-failure

[Install]
WantedBy=multi-user.target

Enable the new service:

# systemctl enable udmg-auth-proxy.service
Created symlink /etc/systemd/system/multi-user.target.wants/udmg-auth-proxy.service → /etc/systemd/system/udmg-auth-proxy.service.

Start the service and check the status:

# systemctl start udmg-auth-proxy
# systemctl status udmg-auth-proxy
● udmg-auth-proxy.service - UDMG Auth Proxy server
Loaded: loaded ( /etc/systemd/system/udmg-auth-proxy.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2022-06-07 16:58:48 -03; 21s ago
Main PID: 25008 (udmg-auth-proxy)
Tasks: 3 (limit: 3509)
CPU: 4ms
CGroup: /system.slice/udmg-auth-proxy.serviceservice
└─25008 /opt/udmg/bin/udmg-auth-proxy

Make sure that the listen port and network interface is reachable by NGINX Server.

UDMG Agent Proxy

Agent Proxy Server Service

Create a new service definition:

# vi /etc/systemd/system/udmg-agent-proxy-server.service

[Unit]

Description=UDMG Agent Proxy server

[Service]
Type=simple
User=udmg
Group=udmg
WorkingDirectory=/home/udmg
Environment="UDMG_AGENT_PROXY_CONFIG=/opt/udmg/etc/udmg/agent/server.toml"
ExecStart=/bin/sh -c 'exec /opt/udmg/bin/udmg-agent-proxy-server'
Restart=on-failure

[Install]
WantedBy=multi-user.target

Enable the new service:

# systemctl enable udmg-agent-proxy-server
Created symlink /etc/systemd/system/multi-user.target.wants/udmg-agent-proxy-server.service → /etc/systemd/system/udmg-agent-proxy-server.service.

Start the service and check the status:

# systemctl start udmg-agent-proxy-server
# systemctl status udmg-agent-proxy-server
● udmg-agent-proxy-server.service - UDMG Agent Proxy Server
Loaded: loaded ( /etc/systemd/system/udmg-agent-proxy-server.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2022-06-07 16:26:53 -03; 2s ago
Main PID: 25444 (udmg-agent-proxy-server)
Tasks: 5 (limit: 3509)
CPU: 5ms
CGroup: /system.slice/udmg-agent-proxy-server.service
└─25444 /opt/udmg/bin/udmg-agent-proxy-server

Jun 07 16:26:53 localhost.localdomain systemd[1]: Started UDMG Agent Proxy Server.
Jun 07 16:26:53 localhost.localdomain sh[25444]: level=info TS=2022-06-07T19:26:53.624296821Z HostKey=Ok Path=/data/agent

Be sure that the listen port and network interface is reachable by UDMG Agent Client .

Agent Proxy Client Service

Create a new service definition:

# vi /etc/systemd/system/udmg-agent-proxy-client.service

[Unit]
Description=UDMG Agent Proxy Client

[Service]
Type=simple
User=udmg
Group=udmg
WorkingDirectory=/home/udmg
Environment="UDMG_AGENT_PROXY_CONFIG=/opt/udmg/etc/udmg/agent/client.toml"
ExecStart=/bin/sh -c 'exec /opt/udmg/bin/udmg-agent-proxy-client'
Restart=on-failure

[Install]
WantedBy=multi-user.target

Enable the new service:

# systemctl enable udmg-agent-proxy-client.service
Created symlink /etc/systemd/system/multi-user.target.wants/udmg-agent-proxy-client.service → /etc/systemd/system/udmg-agent-proxy-client.service.

Start the service and check the status:

# systemctl start udmg-agent-proxy-client
# systemctl status udmg-agent-proxy-client
● udmg-agent-proxy-client.service - UDMG Agent Proxy Client
Loaded: loaded ( /etc/systemd/system/udmg-agent-proxy-client.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2022-06-07 17:26:53 -03; 2s ago
Main PID: 25445 (udmg-agent-proxy-client)
Tasks: 5 (limit: 3509)
CPU: 6ms
CGroup: /system.slice/udmg-agent-proxy-client.service
└─25445 /opt/udmg/bin/udmg-agent-proxy-client

Jun 07 17:26:53 localhost.localdomain systemd[1]: Started UDMG Agent Proxy Client.
Jun 07 17:26:53 localhost.localdomain sh[25445]: level=info TS=2022-06-07T20:26:53.624296821Z Servers=[]

Ports Configuration

See Network Requirements

Using UDMG with SELinux

Modify the file label so that NGINX (as a process labeled with the httpd_t context) can access the configuration file

# restorecon /etc/nginx/conf.d/*

Modify the file label so that NGINX (as a process labeled with the httpd_t context) can access the asset files

# semanage fcontext -a -t httpd_sys_content_t '/opt/udmg/var/www(/.*)?'
# restorecon -Rv /opt/udmg/var/www

Allow NGINX to reverse proxy through the authentication proxy by setting the httpd_can_network_connect boolean

# setsebool -P httpd_can_network_connect 1

References

This document references the following documents.

Name	Location
Systemd	Understanding and Using Systemd - Linux.com
NGINX with SELinux	Modifying SELinux Settings for Full NGINX and NGINX Plus Functionality
PostgreSQL Client Authentication	21.1. The pg_hba.conf File
PostgreSQL Password Authentication	21.5. Password Authentication

Universal Data Mover Gateway 1.3

UDMG for Linux Installation

Pre-Installation / Upgrade Backups

Upgrading UDMG Server

Installing and Configuring the Components

UDMG Admin UI

UDMG User setup

UDMG Server

UDMG Authentication Proxy

UDMG Agent Proxy

Agent Proxy Server Configuration

Agent Proxy Client Configuration

Setup the Systemd Services

UDMG Server

UDMG Authentication Proxy

UDMG Agent Proxy

Agent Proxy Server Service

Agent Proxy Client Service

Ports Configuration

Using UDMG with SELinux

References