Tutorial - Download Multiple Files with Wildcard Mode

Owned by Stonebranch (Deactivated)
Last updated: Jun 06, 2024

Overview

Managed File Transfer operations are handled as file transfer requests registered in the system.

A transfer request record is created either when there is a download or an upload initiated by a partner client that is connected to a local UDMG Server or when a UDMG Client is triggering a request on the interfaces: CLI, REST API, UAC UDMG File transfer task (see GET a file from an SFTP server). Usually, a transfer request is associated with a single file, where the filename is provided by the client and the transfer rule determines the path.


A wildcard transfer is a special kind of transfer that can include a wildcard in the file name parameter, it is possible to request the upload or download of multiple files towards a remote partner with this wildcard mode.

When the transfer is executed, it will perform a LIST operation over the file path and look for all the files that match the pattern. Afterward, a new "child" transfer is created for each of the files using the same configuration as the wildcard transfer (rule, direction, partner, account).

Wildcard transfers are supported in transfers where UDMG is acting as a client and only for the SFTP protocol.

Wildcard

For files, wildcards can be used only in:

  • Source file specifications, not destination file specifications. (Under some operating systems, it is possible for * and ? to be valid characters in a file name.  If they appear in a destination file, they are treated as file characters, not as wildcards.)

  • The file name portion of file specifications and not part of the directory.

Types of Wildcards

There are two types of wildcards:

  1. Asterisk (*) wildcard matches zero or more characters, except the character '/'.

  2. Question mark (?) wildcard matches a single character, except the character '/'.

  3. Character range ([abc] or [a-c]) wildcard matches a single character in the range.

Examples

Here are some examples of wildcard matching given the following file names in the source directory:

  • test.txt

  • test1.txt

  • test2.txt

  • test3.txt

  • test4.txt

  • test.bin

  • test1.bin

  • test2.bin

Source File Specification

Files Copied

*, *.*, test*.*, tes?*.*

Matches and copies all of the files in the directory.

*.txt

Matches and copies the first five files.

*.bin

Matches and copies the last three files.

test?.txt

Matches and copies files test1.txt, test2.txt, test3.txt, and test4.txt.

test[2-4].txt

Matches and copies files test2.txt, test3.txt, and test4.txt.

Transfer Info and Metadata

The wildcard transfer, also called "parent" transfer, can be distinguished from a normal transfer because it will have certain transfer info entries:

Name

Value

Example

udmg_xfer_wildcard

Flag indicating a transfer in wildcard mode

true

udmg_xfer_childIDs

List of child transfer request IDs

161, 162

udmg_xfer_files

List of matching file names

winceclient.png, winceclientSmall.png

The "child" transfers can be distinguished from a normal transfer because they will have certain transfer info entries:

Name

Value

Example

udmg_xfer_parentID

Reference to the parent transfer ID

160

Any additional transfer info entry present in the wildcard transfer will be copied to the "child" transfers.

It is set with the flag udmg_xfer_wildcard:true in the custom transfer information fields.

Examples

Example of a command line with the UDMG Client to fetch the files matching the pattern “wince*.png” from the remote partner “rebex” (see Tutorial - Creating a PeSIT Partner).

$ udmg-client transfer add --way receive --partner rebex --login demo --rule rebex_receive_example --info udmg_xfer_wildcard:true --file 'wince*.png'
The transfer of file wince*.png was successfully added with ID 160.


Example of a command line to register the same request on the REST API with the curl tool, notice the transferInfofield:

$ curl -i -L 'http://hostname/api/transfers' -H 'Content-Type: application/json' -d '{
        "rule": "rebex_receive_example",
        "partner": "rebex",
        "account": "demo",
        "isSend": false,
        "file": "wince*.png",
        "transferInfo": { "udmg_xfer_wildcard": true}
}'
HTTP/1.1 201 Created
Location: /api/transfers/160
Date: Wed, 01 Mar 2023 13:57:58 GMT
Content-Length: 0

The result is the creation of a transfer record with the wildcard pattern, which executes as follows:

Step 1

Connect to the remote partner.

Step 2

List the files that are matching with the pattern on the remote path.

Step 3

Create a child transfer record for each of the files. The transfer is set to planned status and is independent of the parent request.

Step 4Terminate and keep the list of files and child requests in its transfer info metadata.
Step 5

Additionally, the file size field is set to the number of matched files (and child requests).

Step 6

The child requests are processed independently of the parent request and can be paused, resumed, or canceled on their own.

In this example, the transfer with ID 160 is the parent request, with the wildcard pattern, while 161 and 162 are the children requests for the 2 matching files: 'winceclient.png' and 'winceclientSmall.png'.


The transfer record metadata is appended with the following fields:

Name

Value

Example

udmg_xfer_wildcard

Flag indicating a transfer in wildcard mode, only for parent.

true

udmg_xfer_childIDs

List of child transfer request IDs, only for parent.

161, 162

udmg_xfer_files

List of matching file names, only for parent.

winceclient.png, winceclientSmall.png

udmg_xfer_parentID

Reference to the parent transfer ID, only for child.

160

Example of transfer metadata for the parent, on the Admin UI and from the command line:

$ udmg-client transfer get 160
● Transfer 160 (receive as client) [DONE]
    Remote ID: 1630930379848286208
    Protocol: sftp
    Rule: rebex_receive_example
    Requester: demo
    Requested: rebex
    Local filepath: /data/tmp/wince*.png
    Remote filepath: /pub/example/wince*.png
    File size: 2
    Start date: 2023-03-01 13:57:58.434889 +0000 UTC
    End date: 2023-03-01 13:58:07.030964 +0000 UTC
    Bytes transferred: 0
    Transfer values:
        - udmg_sftp_client_version: SSH-2.0-Go
        - udmg_sftp_config_ciphers: [aes128-gcm@openssh.com chacha20-poly1305@openssh.com aes128-ctr aes192-ctr aes256-ctr]
        - udmg_sftp_config_kex: [curve25519-sha256@libssh.org ecdh-sha2-nistp256 ecdh-sha2-nistp384 ecdh-sha2-nistp521 diffie-hellman-group-exchange-sha256 diffie-hellman-group1-sha1 diffie-hellman-group14-sha1]
        - udmg_sftp_config_mac: [hmac-sha2-256-etm@openssh.com hmac-sha2-256 hmac-sha1 hmac-sha1-96]
        - udmg_sftp_failed_auth_methods: [none]
        - udmg_sftp_hostkey_algo: ssh-rsa
        - udmg_sftp_kex_algo: curve25519-sha256@libssh.org
        - udmg_sftp_read_algos: map[cipher:aes128-gcm@openssh.com compression:none mac:]
        - udmg_sftp_server_version: SSH-2.0-RebexSSH_5.0.8208.0
        - udmg_sftp_sucessful_auth_methods: password
        - udmg_sftp_write_algos: map[cipher:aes128-gcm@openssh.com compression:none mac:]
        - udmg_xfer_childIDs: [161 162]
        - udmg_xfer_files: [winceclient.png winceclientSmall.png]
        - udmg_xfer_log: /config/logs/160.log
        - udmg_xfer_wildcard: true

Example of transfer metadata for the child, on the Admin UI and from the command line:


$ udmg-client transfer get 161
● Transfer 161 (receive as client) [DONE]
    Remote ID: 1630930415730556928
    Protocol: sftp
    Rule: rebex_receive_example
    Requester: demo
    Requested: rebex
    Local filepath: /data/in/winceclient.png
    Remote filepath: /pub/example/winceclient.png
    File size: 2635
    Start date: 2023-03-01 13:57:58.434889 +0000 UTC
    End date: 2023-03-01 13:58:17.005211 +0000 UTC
    Bytes transferred: 2635
    Transfer values:
        - udmg_sftp_client_version: SSH-2.0-Go
        - udmg_sftp_config_ciphers: [aes128-gcm@openssh.com chacha20-poly1305@openssh.com aes128-ctr aes192-ctr aes256-ctr]
        - udmg_sftp_config_kex: [curve25519-sha256@libssh.org ecdh-sha2-nistp256 ecdh-sha2-nistp384 ecdh-sha2-nistp521 diffie-hellman-group-exchange-sha256 diffie-hellman-group1-sha1 diffie-hellman-group14-sha1]
        - udmg_sftp_config_mac: [hmac-sha2-256-etm@openssh.com hmac-sha2-256 hmac-sha1 hmac-sha1-96]
        - udmg_sftp_failed_auth_methods: [none]
        - udmg_sftp_hostkey_algo: ssh-rsa
        - udmg_sftp_kex_algo: curve25519-sha256@libssh.org
        - udmg_sftp_read_algos: map[cipher:aes128-gcm@openssh.com compression:none mac:]
        - udmg_sftp_server_version: SSH-2.0-RebexSSH_5.0.8208.0
        - udmg_sftp_sucessful_auth_methods: password
        - udmg_sftp_write_algos: map[cipher:aes128-gcm@openssh.com compression:none mac:]
        - udmg_xfer_log: /config/logs/161.log
        - udmg_xfer_parentID: 160