Disclaimer

Your use of this download is governed by Stonebranch Integration Hub - Terms of Use.

Overview

UDMG (Universal Data Mover Gateway) is a business-to-business MFT (Managed File Transfer) Solution. It provides the capabilities to securely manage File transfers with external business partners (i.e. Customers or Suppliers) and between any endpoint in today’s Hybrid-IT infrastructure, on-premises, or in the cloud (public and or private).

This Universal Extension provides the capability to perform file transfer where UDMG is the client and upload or download files to remote partners.

Version Information

Refer to the changelog for version history information.

Key Features

Software Requirements

This integration requires a Universal Agent and a Python runtime to execute the Universal Task.

Configuration Examples

Scenario A Example for sending a file to a HTTPS server.	Scenario A The partner configuration on UDMG.
Scenario B Example of downloading a file from an SFTP server.	Scenario B The partner configuration on UDMG.
Scenario C Example of downloading multiple files from an SFTP server.	Scenario D Example of listing files from an SFTP server.

Action Output

{
    "exit_code": 0,
    "status_description": "Task executed successfully",
    "invocation": {
        "fields": {
            "credentials_user": "admin",
            "credentials_password": "****",
            "action": "GET",
            "protocol": "sftp",
            "mft_server": "http://udmg.somehost.com:8080/api",
            "file": "readme.txt",
            "output": null,
            "partner": "rebex",
            "rule": "rebex_receive_example",
            "use_dynamic_partner_account": false,
            "partner_account_dynamic": null,
            "user": "demo",
            "wait_for_completion": true,
            "transfer_date": null,
            "polling_interval": 5,
            "max_file_to_monitor": 100,
            "ue_task_uuid": "16835525405027449004GAT04TVODUWH"
        }
    },
    "result": {
        "mft_transfer": {
            "transfer_id": "478",
            "start": "2023-05-11 10:04:36.433385+00:00",
            "stop": "2023-05-11 10:04:40.908049+00:00",
            "status": "DONE",
            "step": "StepNone",
            "transfer_progress": "405  Bytes",
            "task_number": null,
            "error_code": "TeOk",
            "error_msg": null,
            "local_file": "/data/in/readme.txt",
            "remote_file": "/pub/example/readme.txt",
            "transfer_uuid": "1656601189657083904",
            "transfer_info": {
                "udmg_file_computed_extension": ".txt",
                "udmg_file_computed_mimetype": "text/plain; charset=utf-8",
                "udmg_sftp_client_version": "SSH-2.0-Go",
                "udmg_sftp_config_ciphers": [
                    "aes128-gcm@openssh.com",
                    "chacha20-poly1305@openssh.com",
                    "aes128-ctr",
                    "aes192-ctr",
                    "aes256-ctr"
                ],
                "udmg_sftp_config_kex": [
                    "curve25519-sha256@libssh.org",
                    "ecdh-sha2-nistp256",
                    "ecdh-sha2-nistp384",
                    "ecdh-sha2-nistp521",
                    "diffie-hellman-group-exchange-sha256",
                    "diffie-hellman-group1-sha1",
                    "diffie-hellman-group14-sha1"
                ],
                "udmg_sftp_config_mac": [
                    "hmac-sha2-256-etm@openssh.com",
                    "hmac-sha2-256",
                    "hmac-sha1",
                    "hmac-sha1-96"
                ],
                "udmg_sftp_failed_auth_methods": [
                    "none"
                ],
                "udmg_sftp_hostkey_algo": "ssh-rsa",
                "udmg_sftp_kex_algo": "curve25519-sha256@libssh.org",
                "udmg_sftp_read_algos": {
                    "cipher": "aes128-gcm@openssh.com",
                    "compression": "none",
                    "mac": ""
                },
                "udmg_sftp_server_version": "SSH-2.0-RebexSSH_5.0.8466.0",
                "udmg_sftp_sucessful_auth_methods": "password",
                "udmg_sftp_write_algos": {
                    "cipher": "aes128-gcm@openssh.com",
                    "compression": "none",
                    "mac": ""
                },
                "udmg_ue_task_uuid": "16835525405027449004GAT04TVODUWH",
                "udmg_xfer_log": "/config/logs/478.log"
            }
        },
        "mft_transfer_files_summary": {
            "done": 1,
            "error": 0,
            "cancelled": 0
        },
        "errors": []
    }
}

{
    "exit_code": 0,
    "status_description": "Task executed successfully",
    "invocation": {
        "fields": {
            "credentials_user": "admin",
            "credentials_password": "****",
            "action": "MGET",
            "protocol": "sftp",
            "mft_server": "http://udmg.somehost.com:8080/api",
            "file": "m*.png",
            "output": null,
            "partner": "rebex",
            "rule": "rebex_receive_example",
            "use_dynamic_partner_account": false,
            "partner_account_dynamic": null,
            "user": "demo",
            "wait_for_completion": true,
            "transfer_date": null,
            "polling_interval": 5,
            "max_file_to_monitor": 100,
            "ue_task_uuid": "168355254050277290080KCA7YT971DJ"
        }
    },
    "result": {
        "mft_transfer": {
            "transfer_id": "479",
            "start": "2023-05-11 10:44:30.493303+00:00",
            "stop": "2023-05-11 10:44:31.144255+00:00",
            "status": "DONE",
            "step": "StepNone",
            "transfer_progress": "3 out of 3 files",
            "task_number": null,
            "error_code": "TeOk",
            "error_msg": null,
            "local_file": "/data/in/m*.png",
            "remote_file": "/pub/example/m*.png",
            "transfer_uuid": "1656611231072518144",
            "transfer_info": {
                "udmg_sftp_client_version": "SSH-2.0-Go",
                "udmg_sftp_config_ciphers": [
                    "aes128-gcm@openssh.com",
                    "chacha20-poly1305@openssh.com",
                    "aes128-ctr",
                    "aes192-ctr",
                    "aes256-ctr"
                ],
                "udmg_sftp_config_kex": [
                    "curve25519-sha256@libssh.org",
                    "ecdh-sha2-nistp256",
                    "ecdh-sha2-nistp384",
                    "ecdh-sha2-nistp521",
                    "diffie-hellman-group-exchange-sha256",
                    "diffie-hellman-group1-sha1",
                    "diffie-hellman-group14-sha1"
                ],
                "udmg_sftp_config_mac": [
                    "hmac-sha2-256-etm@openssh.com",
                    "hmac-sha2-256",
                    "hmac-sha1",
                    "hmac-sha1-96"
                ],
                "udmg_sftp_failed_auth_methods": [
                    "none"
                ],
                "udmg_sftp_hostkey_algo": "ssh-rsa",
                "udmg_sftp_kex_algo": "curve25519-sha256@libssh.org",
                "udmg_sftp_read_algos": {
                    "cipher": "aes128-gcm@openssh.com",
                    "compression": "none",
                    "mac": ""
                },
                "udmg_sftp_server_version": "SSH-2.0-RebexSSH_5.0.8466.0",
                "udmg_sftp_sucessful_auth_methods": "password",
                "udmg_sftp_write_algos": {
                    "cipher": "aes128-gcm@openssh.com",
                    "compression": "none",
                    "mac": ""
                },
                "udmg_ue_task_uuid": "168355254050277290080KCA7YT971DJ",
                "udmg_xfer_childIDs": [
                    480,
                    481,
                    482
                ],
                "udmg_xfer_files": [
                    "mail-editor.png",
                    "mail-send-winforms.png",
                    "mime-explorer.png"
                ],
                "udmg_xfer_log": "/config/logs/479.log",
                "udmg_xfer_wildcard": true
            }
        },
        "mft_transfer_files_summary": {
            "done": 3,
            "error": 0,
            "cancelled": 0
        },
        "errors": []
    }
}

{
    "exit_code": 0,
    "status": "SUCCESS",    
    "status_description": "SUCCESS",
    "invocation": {
        "fields": { ... }
    },
    "result": {
        "mft_transfer": {
            "transfer_id": "145",
            "start": "2023-05-26T12:36:36.405685+00:00",
            "stop": "2023-05-26T12:36:37.022167",
            "status": "DONE",
            "step": "StepNone",
            "progress": 2,
            "task_number": null,
            "error_code": "TeOk",
            "error_msg": null,
            "local_file": "/home/udmg/udmg-server/in/wince*.png",
            "remote_file": "/pub/example/wince*.png",
            "transfer_uuid": "1662075259576123392",
            "transfer_info": {
                "udmg_sftp_hostkey_algo": "ssh-rsa",
                "udmg_sftp_kex_algo": "curve25519-sha256@libssh.org",
                "udmg_sftp_read_algos": {
                    "cipher": "aes128-gcm@openssh.com",
                    "compression": "none",
                    "mac": ""
                },
                "udmg_sftp_server_version": "SSH-2.0-RebexSSH_5.0.8466.0",
                "udmg_sftp_sucessful_auth_methods": "password",
                "udmg_sftp_write_algos": {
                    "cipher": "aes128-gcm@openssh.com",
                    "compression": "none",
                    "mac": ""
                },
                "udmg_xfer_files": [
					"winceclient.png",
					"winceclientSmall.png"
                ],                
                "udmg_xfer_log": "/home/mft-user/waarp/log/145.log",
                "udmg_xfer_wildcard": true,
                "udmg_xfer_cmd": "list",
                "udmg_ue_task_uuid": "8a801e6d164c4648a32f94dbcd1bb19a"
            }
        }
    }
}

The Extension output contains the attribute result which contains the following sub-attributes:

Input Fields

Dynamic Commands

Cancelation

When the task instance is canceled on the Controller, a cancel command is issued on the UDMG transfer to attempt a graceful termination.

• If the transfer has no children (PUT, GET, LIST action):

  • Fetch the UDMG transfer status and update the output-only fields.

  • Proceed to cancel the UDMG transfer only when its status is one of PLANNED, RUNNING, INTERRUPTED, PAUSED, or ERROR.

  • Update output only fields when the operation is completed.

• If the transfer has children (MPUT, MGET action): cancel the execution of all of the eligible child transfers on UDMG ( in status PLANNED, RUNNING, INTERRUPTED, PAUSED, ERROR). Failure on one transfer does not block the execution of the other transfers.

Re-Run

When the task instance is re-run on the Controller, a resume command is issued on the UDMG transfer for the file transfers in status ERROR.

• If the transfer has no children (PUT, GET, LIST action):

  • Fetch the UDMG transfer status and update the output-only fields.

  • Proceed to resume the UDMG transfer only when its status is ERROR, otherwise

  • Continue the monitoring until the UDMG transfer is completed.

• If the transfer has children (MPUT, MGET action): resume the execution of all of the eligible child transfers on UDMG (in status  ERROR). Failure on one transfer does not block the execution of the other transfers.

Task Output

Output Only Fields

These fields are also available as task instance variables, with the ops_mft_ prefix:

UDMG Transfer Error Code

Exit Codes

The exit codes for this Universal Extension are described below.

STDOUT and STDERR

STDERR provides additional information to the user about the task instance execution. Verbosity is increased with the Task Log Level, The Debug and Trace levels can lead to extensive details and are only recommended for troubleshooting.

STDOUT provides a summary report for the wildcard transfers, but only if Wait For Transfer Completion is set.

For MPUT and MGET actions, the list of the child file transfers with the following details.

  id  status    localFilepath                    start                             stop                                filesize    progress             remoteID  errorCode    errorMsg    step      taskNumber    remoteFilepath                       rule                   isSend    requested    requester
----  --------  -------------------------------  --------------------------------  --------------------------------  ----------  ----------  -------------------  -----------  ----------  --------  ------------  -----------------------------------  ---------------------  --------  -----------  -----------
 480  DONE      /data/in/mail-editor.png         2023-05-11 10:44:30.493303+00:00  2023-05-11 10:44:40.853704+00:00       16471       16471  1656611233584906240  TeOk                     StepNone                /pub/example/mail-editor.png         rebex_receive_example  False     rebex        demo
 481  DONE      /data/in/mail-send-winforms.png  2023-05-11 10:44:30.493303+00:00  2023-05-11 10:44:50.856191+00:00       35414       35414  1656611233622654976  TeOk                     StepNone                /pub/example/mail-send-winforms.png  rebex_receive_example  False     rebex        demo
 482  DONE      /data/in/mime-explorer.png       2023-05-11 10:44:30.493303+00:00  2023-05-11 10:45:00.860761+00:00       49011       49011  1656611233656209408  TeOk                     StepNone                /pub/example/mime-explorer.png       rebex_receive_example  False     rebex        demo

For a LIST action, the list of matching files is presented: 

Filename
------------------------------------------------
winceclient.png
winceclientSmall.png

Configure Universal Task

For a new Universal Task, create a new task, and enter the required input fields.

Integration Modifications

Modifications applied by users or customers, before or after import, might affect the supportability of this integration. The following modifications are discouraged to retain the support level as applied for this integration.

• Python code modifications should not be done.

• Template Modifications

  • General Section

    • "Name", "Extension", "Variable Prefix", and "Icon" should not be changed.

  • Universal Template Details Section

    • "Template Type", "Agent Type", "Send Extension Variables", and "Always Cancel on Force Finish" should not be changed.

  • Result Processing Defaults Section

    • Success and Failure Exit codes should not be changed.

    • Success and Failure Output processing should not be changed.

  • Fields Restriction Section
    The template setup does not impose any restrictions concerning the "Exit Code Processing Fields" section.

    1. Success/Failure exit codes need to be respected.

    2. In principle, as STDERR and STDOUT outputs can change in follow-up releases of this integration, they should not be considered reliable sources for determining the success or failure of a task.

Users and customers are encouraged to report defects, or feature requests at Stonebranch Support Desk.

Document References

This document references the following documents:

Changelog

ue-mft-transfer-1.4.1 (2024-07-25)

Maintenance Changes

• Changed: Refactored the extension (#40864).

• Changed: Documentation update: added compatibility with Python 3.11 with Universal Agent 7.6 (#41489)

Fixes

• Fixed: Error "Partner Account: Field is required" when saving a task after selecting "Dynamic Retrieval of Partner Account Options" on Universal Controller 7.6 (#41543)

ue-mft-transfer-1.4.0 (2024-06-06)

Deprecations and Breaking Changes

• Deprecation: Integration is now compatible with Universal Controller and Universal Agent of version 7.3 and up.

Enhancements

• Added: Increased the maximum number of retrieved values for dynamic commands from 20 to 1000 (#36860).

• Added: Support for dynamic retrieval of partner accounts during task configuration (#32194).
• Added: Enhanced logging on DEBUG log level (#32506).
  
• Changed: The Transfer ID output field is now populated instantly (#35768).

• Changed: API endpoints that support retrieval of entities based on Business Services (introduced on UDMG 1.2) are now used (#35581).

• Changed: When using UMDG 2.0 and up, the dynamic retrieval of rules is now based on the account (#36832).

Fixes

• Fixed: The environment variable `UDMG_CLIENT_MAX_THROTTLE_RATE` is now correctly interpreted as a float (#34325).
• Fixed: Corrected the exit code of tasks when some transfers were manually canceled. The extension now correctly returns exit code 100 (#34316).

ue-mft-transfer-1.3.1 (2024-03-12)

Enhancements

• Added: Compatibility with Python 3.11 (#35935).

ue-mft-transfer-1.3.0 (2023-09-29)

Deprecations and Breaking Changes

• Breaking Change: "Polling Interval" minimum value is set to 1, a value of 0 could cause an infinite loop (#33815).

Enhancements

• Added: new LIST action to retrieve the names of the files matching a wildcard pattern on a remote SFTP partner (#33358).
• Added: Logging of dynamic command processing in the agent log with TRACE level (#32993).
• Added: MGET/MPUT: error is raised when monitoring of child transfers is skipped because the list is greater than "Max Files To Monitor": CHILD_TRANSFERS_NOT_MONITORED_WARNING with exit code 90 (#33065).
• Added: Support for throttling the MGET/MPUT monitoring with the environment variable UDMG_CLIENT_MAX_THROTTLE_RATE. It sets the maximum number of calls per second to the UDMG API during the monitoring phase. Default is 60 (#33432).
• Changed: "Target File" only visible for GET & PUT actions (#33367).
• Changed: After a dynamic command, the extension will terminate immediately if the transfer is already completed (#32989).
• Changed: "Polling Interval" field is always visible, as it is still used to check for parent transfer completion (#32994).
• Changed: MGET/MPUT: "Transfer Progress" is only updated when a child transfer is completed (DONE, ERROR, CANCELLED) (#33815).
• Changed: Sorting of the "Action" list (#33064).

Fixes

• Fixed: Exit code to be 11 in case of 404 API errors (#28881).
• Fixed: Extension finishes successfully after unexpected transfer CANCELLED status (#29248).
• Fixed: Fix Task Cancel to also process child transfers that are in ERROR status (#33804).
• Fixed: Proper catching of UDMG API errors during wildcard transfer monitoring (#33310).

ue-mft-transfer-1.2.0 (2023-05-09)

Deprecations and Breaking Changes

• Breaking Change: Transfer Progress output only field is changed to text value (#32195)

Enhancements

• Add: Support Wildcard Transfer (#32198, #32195, #32201)
  
  1. New actions "MPUT" and "MGET" for wildcard transfers:
     • support for wildcard in source file parameter,
     • creation of wildcard transfer on UDMG,
     • monitoring of child transfers on UDMG,
     • tabular report on STDOUT with final child transfer status,
     • short summary report in EXTENSION output,
     • PAUSE, RESUME, and CANCEL commands are propagated to child transfers
  2. "Transfer Progress" output field is redefined to show human readable value
     • size in KB, MB, or GB for single transfer,
     • "x out of y files" for wildcard transfer
  3. "Max files to monitor" parameter: limit the number of child transfers that are monitored by the task instances, 100 by default.
  4. "Wait for transfer completion" parameter: task instance only returns when the transfer is completed (with DONE or CANCELLED status) on UDMG.
     For wildcard transfers, only returns when all child transfers are completed. Activated by default.
• Add: Refresh task instance output with re-run (#32463)
• Add: Keep UC task instance ID in UDMG file transfer metadata field udmg_ue_task_uuid (#30931)

ue-mft-transfer-1.1.0 (2023-01-31)

Deprecations and Breaking Changes

• Breaking Change: minimum universal agent version set to 7.3

Enhancements

• Add: set target filename (rename file) (#31174)
• Add: transfer status as UC 7.3 extension status field. (#31041)
• Add: retry failed transfer with Task Re-run. (#29599)
• Add: show UUID. (#30733)
• Add: show transfer info fields. (#30930)
• Add: support for PeSIT and PeSIT-TLS protocols (#31051)

Fixes

• Fix: optimize the number of API calls. (#29247)

ue-mft-transfer-1.0.0 (2022-06-30)

• Initial version