UDMG for Linux Installation
For the UDMG Web Transfer Client, please refer to UDMG Web Transfer Client for Linux Installation.
Upgrading Universal Data Mover Gateway
Upgrading UDMG refers to the increase of a currently installed pre-1.4.x Version, Release, or Modification level of UDMG (1.3.x, 1.2.x, 1.1.x, 1.0.x) to UDMG 1.4.x.
Note
If you are upgrading an installation of UDMG from any release prior to 1.3.0.0, you must uninstall the older version before installmysqling the new version.
The installation packages, binaries, services, and environment variables have changed and this does not allow for a standard upgrade.
| Category | Prior releases | Release 1.3 |
|---|---|---|
| user and group | mft:mft | udmg:udmg |
| binaries | /usr/local/bin | /opt/udmg/bin |
| configuration files | /etc/mft | /opt/udmg/etc |
| log files | /var/opt/udmg/logs | /var/opt/udmg/logs |
| UDMG Admin UI assets | /opt/udmg/var/www/mft | /opt/udmg/var/www/udmg |
| Services |
|
|
| Environment variables |
|
|
It may be required to modify the work and data directories ownership or access rights and to update UDMG Server transfer rules to use paths that are accessible by the 'udmg' user.
The configuration files must be reviewed and compared between the old and new locations.
Special attention is required for the AESpassphrase parameter for UDMG Server. It must be the path for the file that was used by the previous release and must be accessible by the new service user. It is recommended to set an absolute path in the configuration file.
Pre-Installation / Upgrade Backups
The installation process overwrites the current files (exception: the configuration files are kept), this may affect your modifications.
Backing up the configuration files will optimize the time it takes you to get up and running after installing or upgrading.
/opt/udmg/etc/udmg/nginx/udmg.conf/opt/udmg/etc/udmg/agent/client.toml/opt/udmg/etc/udmg/agent/server.toml/opt/udmg/etc/udmg/auth-proxy/config.toml/opt/udmg/etc/udmg/web-transfer/config.toml/opt/udmg/etc/udmg-server/server.ini
After upgrading RPM or DEB packages, review the new configuration file templates (with the extension .rpmnew or .dpk-new) and edit the current configuration files to add new parameters or remove deprecated parameters.
Release migration
The UDMG release version is stored in the database to ensure the data structure is compatible with the version of the UDMG components.
After upgrading the component binaries and before starting the UDMG server it is required to perform the release migration step.
The udmg-server "migrate" command handles the necessary database updates and the setting of the internal version.
$ /opt/udmg/bin/udmg-server migrate --help
Usage:
udmg-server [OPTIONS] migrate [migrate-OPTIONS] [version]
Help Options:
-h, --help Show this help message
[migrate command options]
-c, --config= The configuration file to use
-d, --dry-run Simulate the migration but does not commit the changes
-l, --list List Migrations
-f, --file= Writes the migration commands into a file instead of sending them to the database
-v, --verbose Show verbose debug information. Can be repeated to increase verbosity
[migrate command arguments]
version: The version to which the database should be migrated
The configuration file is the one that will be used for the server mode, with the parameter for accessing the target database.
To get the list of supported target version, use the list parameter. The last version is the release version of the udmg-server.
$ /opt/udmg/bin/udmg-server migrate -c /opt/udmg/etc/udmg-server/server.ini --list | tail -n 3 1.3.0 1.4.0 1.4.1
It is recommended to set the verbose parameter (3 times) to follow the progress.
$ /opt/udmg/bin/udmg-server migrate -c /opt/udmg/etc/udmg-server/server.ini -vvv 1.4.1 [INFO ] Migration: Starting upgrade migration... [INFO ] Migration: Applying migration 'Bump database version to 1.4.1'
After the migration, the services for UDMG components can be started.
Upgrading Universal Data Mover Gateway for Linux
Upgrading with Linux software packages
Step 1 | Contact your Stonebranch representative or the Customer Support to receive the software package for the intended operating system. |
|---|---|
Step 2 | Perform the recommended backup of configuration files. |
Step 3 | Stop the components services. The exact steps depend on the system architecture and the deployed components, for example:
|
Step 4 | Upgrade the UDMG packages (RPM or DEB), for example:
|
| Step 5 | Review the component configuration files. Note that new configuration file templates (with the extension .rpmnew or .dpk-new) that contains all the allowed parameters are added during the software package upgrade. |
| Step 6 | Perform the release migration. |
| Step 7 | Start the components services. The exact steps depend on the system architecture and the deployed components, for example:
|
Upgrading a manual installation
Step 1 | Contact your Stonebranch representative or the Customer Support to receive the software package for the intended operating system. |
|---|---|
Step 2 | Perform the recommended backup of configuration files. |
Step 3 | Stop the components services. The exact steps depend on the system architecture and the deployed components, for example:
|
Step 4 | Replace the component binaries
Change ownership/permissions on new files:
Upgrade the Admin UI:
|
| Step 5 | Review the component configuration files. Refer to each component installation section below for the list of parameters. |
| Step 6 | Perform the release migration. |
| Step 7 | Start the components services. The exact steps depend on the system architecture and the deployed components, for example:
|
Installing and Configuring the Components
Installing with Linux software packages
Step 1 | Contact your Stonebranch representative or the Customer Support to receive the software package for the intended operating system. |
|---|---|
Step 2 | Install the UDMG packages (RPM or DEB), for example:
|
Step 3 | Review the component configuration files. |
Step 4 | Start the components services. The exact steps depend on the system architecture and the deployed components, for example:
|
Performing a manual installation
UDMG Admin UI
The following steps require root privilege, make sure that you have the correct access before continuing.
Extract the distribution file for UDMG Admin UI, under the directory web server root directory, see the NGINX Server configuration.
# unzip -d /opt/udmg/var/www/udmg/ udmg-admin-ui-<VERSION>.zipsudo
- The zip file can now be deleted.
- Validate that the service is working properly with the 'curl' command:
# curl http://localhost:80 -IHTTP/1.1 200 OKServer: nginx/1.21.6Date: Mon, 06 Jun 2022 17:33:19 GMTContent-Type: text/htmlContent-Length: 7788Last-Modified: Fri, 03 Jun 2022 14:07:05 GMTConnection: keep-aliveETag: "629a1589-1e6c"Accept-Ranges: bytes
UDMG User setup
Create a dedicated user for running the UDMG modules and to be the owner of the files that will be transferred by UDMG.
# groupadd udmgsudo
# useradd -g udmg udmgsudo
UDMG Server
Create the configuration file /opt/udmg/etc/udmg-server/server.ini with the following parameters:
# mkdir -p /opt/udmg/etc/udmg-serversudo # vi /opt/udmg/etc/udmg-server/server.inisudo
Note
The lines starting with a semicolon ';' or a hash '#' are comments, describing the option or showing the default value.
The parameters must be adapted to your environment, in particular:
- global section: GatewayHome
- log section: LogLevel, LogTo, LogPath
- admin section: Host, Port
- database section: Type, Address, Name, User, Password
[global]; The name given to identify this gateway instance. If the the database is shared between multiple gateways, this name MUST be unique across these gateways.GatewayName = udmg
; Default OS permission for created files; FilePermissions = 770
; Default OS permission for created directories; DirPermissions = 770
[paths]; The root directory of the gateway. By default, it is the working directory of the process.GatewayHome = /home/udmg
; The directory for all incoming files.; DefaultInDir = in
; The directory for all outgoing files.; DefaultOutDir = out
; The directory for all running transfer files.; DefaultTmpDir = tmp
[log]; All messages with a severity above this level will be logged. Possible values are TRACE, DEBUG, INFO, WARNING, ERROR and CRITICAL.Level = DEBUG
; The path to the file where the logs must be written. Special values 'stdout' and 'syslog' log respectively to the standard output and to the syslog daemon.; LogTo = stdout
; If LogTo is set on 'syslog', the logs will be written to this facility.; SyslogFacility = local0
; The directory for the log files of the local servers, partners, and transfers. No default, if not provided then the detailed log feature is disabled.
; LogPath = /var/opt/udmg/logs
[admin]; The address used by the admin interface.Host = 0.0.0.0
; The port used by the admin interface. If the port is 0, a free port will automatically be chosen.Port = 18080
; Path of the TLS certificate for the admin interface.; TLSCert =
; Path of the key of the TLS certificate.; TLSKey =
; API rate limiter: number of allowed requests per client IP, per second. After that HTTP code 429 is returned. Disabled if 0 or not provided.
; RateLimit = 0
[database]; Name of the RDBMS used for the gateway database. Possible values: sqlite, mysql, postgresql, oracle, mssqlType = postgresql
; Address of the databaseAddress = localhost
; The name of the databaseName = udmg
; The name of the gateway database userUser = udmg_user
; The password of the gateway database userPassword = udmg_password
; Path of the database TLS certificate file.; TLSCert =
; Path of the key of the TLS certificate file.; TLSKey =
; The path to the file containing the passphrase used to encrypt account passwords using AES; AESPassphrase = passphrase.aes
[controller]; The frequency at which the database will be probed for new transfers; Delay = 5s
; The maximum number of concurrent incoming transfers allowed on the gateway (0 = unlimited).; MaxTransferIn = 0
; The maximum number of concurrent outgoing transfers allowed on the gateway (0 = unlimited).; MaxTransferOut = 0
[sftp]; Set to true to allow legacy and weak cipher algorithms: 3des-cbc, aes128-cbc, arcfour, arcfour128, arcfour256; AllowLegacyCiphers = false
Note
AESPassphrase file is generated on first run if it does not exist. It is recommended to set an absolute path, otherwise it is created in the current directory.
Make sure to verify the file location during an upgrade and to have a backup. Without the correct AESPassphrase file, the passwords, the keys and the certificates will not be usable.
The use of an incorrect passphrase file is reported with the error:
cannot decrypt password: cipher: message authentication failed- Install the binaries under /opt/udmg/bin
:
# install -m 755 udmg-client /opt/udmg/binsudo # install -m 755 udmg-server /opt/udmg/binsudo
UDMG Authentication Proxy
Create a directory under
/etc/udmg/:
# mkdir -p /opt/udmg/etc/udmgsudo
- Create a configuration file for the service:
# vi /opt/udmg/etc/udmg/auth_proxy/config.tomlsudo
# Proxy Configuration[proxy]# Port, default "5000"port = "5000"# Network interface, default "0.0.0.0"inet = "127.0.0.1"# Enable recover on panic, default true, should be true for production environmentrecover = true# Enable Cross-Origin Resource Sharing (CORS), should be true for production environmentcors = true# CORS: List of origins that may access the resource. Optional. Default value "*"# domain = "*"# Enable Request Track ID, default truetracker = true# Enable Request Logger, default truelogger = true# Rate Limit IP Request over 1 second, default 0 (unlimited)limit = 0# Enable the Prometheus Metric Endpoint '/metric', default falsemetrics = false
# Enable CSRF tokencsrf = false# # Service 'local' with direct authentication on the UDMG Server[service.local]# UDMG Server Listen Protocol protocol = "http"[[service.local.targets]]# UDMG Server Hostname or IPhostname = "localhost"# UDMG Server Portport = 18080
Please refer to Authentication Methods for the LDAP and SSO authentication options.
- Install the binary under
/opt/udmg/bin:
# install -m 755 udmg-auth-proxy /opt/udmg/binsudo
UDMG Agent Proxy
Create a directory under /opt/udmg/etc/udmg
:
# mkdir -p /opt/udmg/etc/udmg/agent/sudo
- Install the binaries under
/opt/udmg/bin:
# install -m 755 sudo udmg-agent-client /opt/udmg/bin# install -m 755 sudo udmg-agent-server /opt/udmg/bin
Agent Proxy Server Configuration
Generate a SSH Key for the service:
# ssh-keygen -t rsa -q -N "" -f /opt/udmg/etc/udmg/agent/agent
- Change the agent key permissions:
# chmod 755 /opt/udmg/etc/udmg/agent/agent /opt/udmg/etc/udmg/agent/agent.pub
- Create a configuration file as /opt/udmg/etc/udmg/agent/
agent.toml:
# vi /opt/udmg/etc/udmg/agent/agent.tomlsudo
[agent]# UDMG Agent Proxy Hostname or IP, and porthostname = "0.0.0.0"port = "2222"# path to the SSH private key filessh_key = "/opt/udmg/etc/udmg/agent/agent"# path to the SSH public key filessh_key_pub = "/opt/udmg/etc/udmg/agent/agent.pub"
# Agent Service User and passwordusername = "mft"password = "61ee8b5601a84d5154387578466c8998848ba089"
The password key will be used for the client authentication.
Agent Proxy Client Configuration
Create a configuration file as
/etc/udmg/agent_proxy/client.toml:
# vi /opt/udmg/etc/udmg/agent/client.tomlsudo
[client]# Target UDMG Agent Proxy Hostname or IP, and port hostname = "localhost"port = "2222"
# UDMG Agent Service User and passwordusername = "mft"password = "61ee8b5601a84d5154387578466c8998848ba089"
# Default TTL to Connection Retryttl="5s"
[client.api]# UDMG Agent Client Admin APIport="2280"
[gateway]# UDMG Server Hostname or IP, and port hostname = "localhost"port = "18080"# UDMG Server Username/Passwordusername = "admin"password = "admin_password"
The password key will be used for the client authentication.
Setup the Systemd Services
UDMG Server
Create a new service definition:
# vi /etc/systemd/system/udmg-server.servicesudo
[Unit]
Description=UDMG Server
[Service]
Type=simple
User=udmg
Group=udmg
WorkingDirectory=/home/udmg
ExecStart=/bin/sh -c '/usr/local/bin/udmg-server server -c /opt/udmg/etc/udmg-server/server.ini'
Restart=on-failure
[Install] WantedBy=multi-user.target
- Enable the new service:
# systemctl enable sudo udmg-server.serviceCreated symlink /etc/systemd/system/multi-user.target.wants/udmg-server.service → /etc/systemd/system/udmg-server.service.
- Start the service and check the status:
# systemctl start udmg-server# systemctl status udmg-server● .service - UDMG serverudmg-serverLoaded: loaded ( /etc/systemd/system/udmg-server.service; enabled; vendor preset: disabled)Active: active (running) since Tue 2022-06-07 16:43:16 -03; 10s agoMain PID: 24888 ()udmg-serverTasks: 6 (limit: 3509)CPU: 11msCGroup: /system.slice/udmg-server.service└─24888 /opt/udmg/bin/udmg-server server -c /opt/udmg/etc/udmg-server/server.ini
Make sure that the listen port and network interface is reachable by UDMG Authentication Proxy and UDMG Agent Client.
UDMG Authentication Proxy
Create a new service definition:
# vi /etc/systemd/system/sudo .serviceudmg-auth-proxy
[Unit]Description=UDMG Auth Proxy server
[Service]Type=simpleUser=udmgGroup=udmgWorkingDirectory=/home/udmgEnvironment="UDMG_AUTH_PROXY_CONFIG=/opt/udmg/etc/udmg/auth_proxy/config.toml"ExecStart=/bin/sh -c 'exec /opt/udmg/bin/'udmg-auth-proxyRestart=on-failure
[Install]WantedBy=multi-user.target
- Enable the new service:
# systemctl enable sudo .serviceudmg-auth-proxyCreated symlink /etc/systemd/system/multi-user.target.wants/.service → /etc/systemd/system/udmg-auth-proxy.service.udmg-auth-proxy
- Start the service and check the status:
# systemctl start udmg-auth-proxy# systemctl status udmg-auth-proxy● .service - UDMG Auth Proxy serverudmg-auth-proxyLoaded: loaded ( /etc/systemd/system/.service; enabled; vendor preset: disabled)udmg-auth-proxyActive: active (running) since Tue 2022-06-07 16:58:48 -03; 21s agoMain PID: 25008 (udmg-auth-proxy)Tasks: 3 (limit: 3509)CPU: 4msCGroup: /system.slice/.serviceserviceudmg-auth-proxy└─25008 /opt/udmg/bin/udmg-auth-proxy
Make sure that the listen port and network interface is reachable by NGINX Server.
UDMG Agent Proxy
Agent Proxy Server Service
Create a new service definition:
# vi /etc/systemd/system/udmg-agent-server.servicesudo
[Unit]
Description=UDMG Agent Proxy server
[Service]
Type=simple
User=udmg
Group=udmg
WorkingDirectory=/home/udmg
Environment="UDMG_AGENT_CONFIG=/opt/udmg/etc/udmg/agent/server.toml"
ExecStart=/bin/sh -c 'exec /opt/udmg/bin/udmg-agent-server'
Restart=on-failure
[Install]
WantedBy=multi-user.target
- Enable the new service:
# systemctl enable udmg-agent-serversudo Created symlink /etc/systemd/system/multi-user.target.wants/udmg-agent-server.service → /etc/systemd/system/udmg-agent-server.service.
- Start the service and check the status:
# systemctl start udmg-agent-server# systemctl status udmg-agent-server● udmg-agent-server.service - UDMG Agent Proxy ServerLoaded: loaded ( /etc/systemd/system/udmg-agent-server.service; enabled; vendor preset: disabled)Active: active (running) since Tue 2022-06-07 16:26:53 -03; 2s agoMain PID: 25444 (udmg-agent-server)Tasks: 5 (limit: 3509)CPU: 5msCGroup: /system.slice/udmg-agent-server.service└─25444 /opt/udmg/bin/udmg-agent-server
Jun 07 16:26:53 localhost.localdomain systemd[1]: Started UDMG Agent Proxy Server.Jun 07 16:26:53 localhost.localdomain sh[25444]: level=info TS=2022-06-07T19:26:53.624296821Z HostKey=Ok Path=/data/agent
Be sure that the listen port and network interface is reachable by UDMG Agent Client .
Agent Proxy Client Service
Create a new service definition:
# vi /etc/systemd/system/udmg-agent-client.servicesudo
[Unit]Description=UDMG Agent Proxy Client
[Service]Type=simpleUser=udmgGroup=udmgWorkingDirectory=/home/udmgEnvironment="UDMG_AGENT_CONFIG=/opt/udmg/etc/udmg/agent/client.toml"ExecStart=/bin/sh -c 'exec /opt/udmg/bin/udmg-agent-client'Restart=on-failure
[Install]WantedBy=multi-user.target
- Enable the new service:
# systemctl enable sudo udmg-agent-client.serviceCreated symlink /etc/systemd/system/multi-user.target.wants/udmg-agent-client.service → /etc/systemd/system/udmg-agent-client.service.
- Start the service and check the status:
# systemctl start udmg-agent-client# systemctl status udmg-agent-client● .service - UDMG Agent Proxy Clientudmg-agent-clientLoaded: loaded ( /etc/systemd/system/udmg-agent-client.service; enabled; vendor preset: disabled)Active: active (running) since Tue 2022-06-07 17:26:53 -03; 2s agoMain PID: 25445 (udmg-agent-client)Tasks: 5 (limit: 3509)CPU: 6msCGroup: /system.slice/udmg-agent-client.service└─25445 /opt/udmg/bin/udmg-agent-client
Jun 07 17:26:53 localhost.localdomain systemd[1]: Started UDMG Agent Proxy Client.Jun 07 17:26:53 localhost.localdomain sh[25445]: level=info TS=2022-06-07T20:26:53.624296821Z Servers=[]
Ports Configuration
Using UDMG with SELinux
Security-Enhanced Linux (SELinux) is enabled by default on modern RHEL and CentOS servers. Each operating system object (process, file descriptor, file, etc.) is labeled with an SELinux context that defines the permissions and operations the object can perform. In RHEL 6.6/CentOS 6.6 and later, NGINX is labeled with the httpd_t context.
When SELinux is enabled, the UDMG Admin UI will show "403 access denied" and "404 page not found" errors on the landing page and permission errors are reported in the NGINX log files:
...
2023/09/19 12:51:38 [error] 108236#108236: *1 "/opt/udmg/var/www/udmg/index.html" is forbidden (13: Permission denied), client: 127.0.0.1, server: localhost, request: "GET / HTTP/1.1", host: "localhost
...
- Verify whether SELinux is enforced with the
getenforcecommand. A result of "Enforcing" means that is is enabled and that the steps below are required.
$ getenforceEnforcing
- Modify the file labels so that NGINX (as a process labeled with the
httpd_tcontext) can access the configuration files.
# restorecon /etc/nginx/conf.d/*sudo
- Modify the file labels so that NGINX (as a process labeled with the
httpd_tcontext) can access the UDMG Admin UI asset files.
# semanage fcontext -a -t httpd_sys_content_t '/opt/udmg/var/www(/.*)?'sudo # restorecon -Rv /opt/udmg/var/wwwsudo
- Allow NGINX to reverse proxy through the UDMG Authentication Proxy by setting the
httpd_can_network_connectboolean.
# setsebool -P httpd_can_network_connect 1sudo
References
This document references the following documents.
Name | Location |
|---|---|
Systemd | |
NGINX with SELinux | |
PostgreSQL Client Authentication | |
PostgreSQL Password Authentication |