Tutorial - Download Multiple Files with Wildcard Mode
Overview
Managed File Transfer operations are handled as file transfer requests that are registered in the system.
A transfer request record is created either when there is a download or an upload initiated by a partner client that is connected to a local UDMG server or when a UDMG client is triggering a request on the interfaces: CLI, REST API, UAC UDMG File transfer task (see GET a file from an SFTP server). Usually, a transfer request is associated with a single file, where the filename is provided by the client and the path is determined by the transfer rule.
A wildcard transfer is a special kind of transfer that can include a wildcard in the file name parameter, it is possible to request the upload or download of multiple files towards a remote partner with this wildcard mode.
When the transfer is executed, it will perform a LIST operation over the file path and look for all the files that match the pattern. Afterward, a new "child" transfer is created for each of the files using the same configuration as the wildcard transfer (rule, direction, partner, account).
Wildcard transfers are supported in transfers where UDMG is acting as a client and only for the SFTP protocol.
Wildcard
For files, wildcards can be used only in:
Source file specifications, not destination file specifications. (Under some operating systems, it is possible for * and ? to be valid characters in a file name. If they appear in a destination file, they are treated as file characters, not as wildcards.)
File name portion of file specifications, not as part of the directory.
Types of Wildcards
There are two types of wildcards:
Asterisk (*) wildcard matches zero or more characters, except the character '/'.
Question mark (?) wildcard matches a single character, except the character '/'.
Character range ([abc] or [a-c]) wildcard matches a single character in the range.
Examples
Here are some examples of wildcard matching given the following file names in the source directory:
test.txt
test1.txt
test2.txt
test3.txt
test4.txt
test.bin
test1.bin
test2.bin
Source File Specification | Files Copied |
---|---|
| Matches and copies all of the files in the directory. |
| Matches and copies the first five files. |
| Matches and copies the last three files. |
| Matches and copies files |
| Matches and copies files |
Transfer Info and Metadata
The wildcard transfer, also called "parent" transfer, can be distinguished from a normal transfer because it will have certain transfer info entries:
Name | Value | Example |
---|---|---|
udmg_xfer_wildcard | Flag indicating a transfer in wildcard mode | true |
udmg_xfer_childIDs | List of child transfer request IDs | 161, 162 |
udmg_xfer_files | List of matching file names | winceclient.png, winceclientSmall.png |
The "child" transfers can be distinguished from a normal transfer because they will have certain transfer info entries:
Name | Value | Example |
---|---|---|
udmg_xfer_parentID | Reference to the parent transfer ID | 160 |
Any additional transfer info entry present in the wildcard transfer will be copied to the "child" transfers.
It is set with the flag udmg_xfer_wildcard:true
in the custom transfer information fields.
Examples
Example of a command line with the UDMG client to fetch the files matching the pattern “wince*.png” from the remote partner “rebex” (see Tutorial - Download Multiple Files with Wildcard Mode)
$ waarp-gateway transfer add --way receive --partner rebex --login demo --rule rebex_receive_example --info udmg_xfer_wildcard:true --file 'wince*.png'
The transfer of file wince*.png was successfully added with ID 160.
Example of a command line to register the same request on the REST API with the curl tool, notice the transferInfo
field:
$ curl -i -L 'http://hostname/api/transfers' -H 'Content-Type: application/json' -d '{
"rule": "rebex_receive_example",
"partner": "rebex",
"account": "demo",
"isSend": false,
"file": "wince*.png",
"transferInfo": { "udmg_xfer_wildcard": true}
}'
HTTP/1.1 201 Created
Location: /api/transfers/160
Date: Wed, 01 Mar 2023 13:57:58 GMT
Content-Length: 0
The result is the creation of a transfer record with the wildcard pattern, which executes as follow:
Connect to the remote partner
List the files that are matching with the pattern on the remote path
Create a child transfer record for each of the files. This transfer is set to planned status and is independent from the parent request.
Terminates and keep the list of files and child requests in its transfer info metadata
Additionally, the file size field is set to the number of matched files (and child requests)
The child requests as processed independently from the parent request and can be paused, resumed or cancelled on their own.
In this example, the transfer with ID 160 is the parent request, with the wildcard pattern, while 161 and 162 are the children requests for the 2 matching files: 'winceclient.png' and 'winceclientSmall.png'.
The transfer record metadata is appended with the following fields:
Name | Value | Example |
---|---|---|
udmg_xfer_wildcard | Flag indicating a transfer in wildcard mode, only for parent. | true |
udmg_xfer_childIDs | List of child transfer request IDs, only for parent. | 161, 162 |
udmg_xfer_files | List of matching file names, only for parent. | winceclient.png, winceclientSmall.png |
udmg_xfer_parentID | Reference to the parent transfer ID, only for child. | 160 |
Example of transfer metadata for the parent, on the Admin UI and from the command line:
$ waarp-gateway transfer get 160
● Transfer 160 (receive as client) [DONE]
Remote ID: 1630930379848286208
Protocol: sftp
Rule: rebex_receive_example
Requester: demo
Requested: rebex
Local filepath: /data/tmp/wince*.png
Remote filepath: /pub/example/wince*.png
File size: 2
Start date: 2023-03-01 13:57:58.434889 +0000 UTC
End date: 2023-03-01 13:58:07.030964 +0000 UTC
Bytes transferred: 0
Transfer values:
- udmg_sftp_client_version: SSH-2.0-Go
- udmg_sftp_config_ciphers: [aes128-gcm@openssh.com chacha20-poly1305@openssh.com aes128-ctr aes192-ctr aes256-ctr]
- udmg_sftp_config_kex: [curve25519-sha256@libssh.org ecdh-sha2-nistp256 ecdh-sha2-nistp384 ecdh-sha2-nistp521 diffie-hellman-group-exchange-sha256 diffie-hellman-group1-sha1 diffie-hellman-group14-sha1]
- udmg_sftp_config_mac: [hmac-sha2-256-etm@openssh.com hmac-sha2-256 hmac-sha1 hmac-sha1-96]
- udmg_sftp_failed_auth_methods: [none]
- udmg_sftp_hostkey_algo: ssh-rsa
- udmg_sftp_kex_algo: curve25519-sha256@libssh.org
- udmg_sftp_read_algos: map[cipher:aes128-gcm@openssh.com compression:none mac:]
- udmg_sftp_server_version: SSH-2.0-RebexSSH_5.0.8208.0
- udmg_sftp_sucessful_auth_methods: password
- udmg_sftp_write_algos: map[cipher:aes128-gcm@openssh.com compression:none mac:]
- udmg_xfer_childIDs: [161 162]
- udmg_xfer_files: [winceclient.png winceclientSmall.png]
- udmg_xfer_log: /config/logs/160.log
- udmg_xfer_wildcard: true
Example of transfer metadata for the child, on the Admin UI and from the command line:
$ waarp-gateway transfer get 161
● Transfer 161 (receive as client) [DONE]
Remote ID: 1630930415730556928
Protocol: sftp
Rule: rebex_receive_example
Requester: demo
Requested: rebex
Local filepath: /data/in/winceclient.png
Remote filepath: /pub/example/winceclient.png
File size: 2635
Start date: 2023-03-01 13:57:58.434889 +0000 UTC
End date: 2023-03-01 13:58:17.005211 +0000 UTC
Bytes transferred: 2635
Transfer values:
- udmg_sftp_client_version: SSH-2.0-Go
- udmg_sftp_config_ciphers: [aes128-gcm@openssh.com chacha20-poly1305@openssh.com aes128-ctr aes192-ctr aes256-ctr]
- udmg_sftp_config_kex: [curve25519-sha256@libssh.org ecdh-sha2-nistp256 ecdh-sha2-nistp384 ecdh-sha2-nistp521 diffie-hellman-group-exchange-sha256 diffie-hellman-group1-sha1 diffie-hellman-group14-sha1]
- udmg_sftp_config_mac: [hmac-sha2-256-etm@openssh.com hmac-sha2-256 hmac-sha1 hmac-sha1-96]
- udmg_sftp_failed_auth_methods: [none]
- udmg_sftp_hostkey_algo: ssh-rsa
- udmg_sftp_kex_algo: curve25519-sha256@libssh.org
- udmg_sftp_read_algos: map[cipher:aes128-gcm@openssh.com compression:none mac:]
- udmg_sftp_server_version: SSH-2.0-RebexSSH_5.0.8208.0
- udmg_sftp_sucessful_auth_methods: password
- udmg_sftp_write_algos: map[cipher:aes128-gcm@openssh.com compression:none mac:]
- udmg_xfer_log: /config/logs/161.log
- udmg_xfer_parentID: 160