...
| Template Name | Extension Name | Extension Version | ||||
|---|---|---|---|---|---|---|
| Cloud | File Transferue-cloud-ft | 1.0.0 | CloudData Transfer | ue-cloud-dt | 2.0. | 01 |
Refer to Cloud Data Transfer for version history information.
...
- Create required Resolvable Credentials. Required as Input Fields on the Universal Task Configuration.
- Create a new Script of type Data, for the Configuration File and populate it according to the following section of Setup Rclone Configuration File. Required as Input Fields on the Universal Task Configuration.
- Create a new task, and enter the task-specific details that were created in the Universal Template.
...
The configuration file is following the INI format, and contains all required parameters and credentials to connect to a Storage System. All configuration options per Storage System can be found in the respective "Config Rclone documentation" here.
It is advised that account credentials, tokens, or any other essential information to be passed as an encrypted Resolvable Credential field. To set a value with the respective Credential field, please make sure to use the appropriate Credential function inside the configuration file.
Note 'Runtime Password' Credentials field supports values with up to 512 characters.
Hard coded values can be provided in the script file, as well. However, it is not advised for security reasons.
It is recommended to configure a storage system section, one to one with the Credential fields in this Universal Extension (please refer to the Input Fields section below).
For local file system Storages, no credentials are required in the configuration file. The selected action will be executed with the Credentials that are provided in the "Agent Details" of the Universal Task. If no Credentials are provided for the used Agent, then the same user that runs ubroker daemon will be used.
Script Type should be Data, and option Resolve UAC Variables should be enabled.
...
| Field | Input type | Default value | Type | Description |
|---|---|---|---|---|
| Action | Required | Copy Objects | Choice | Action to be performed. Available actions:
|
| Configuration File | Required | - | Script | Rclone Configuration file, that contains all required parameters to connect to the Storage System, Source Storage and Target Storage. It is indicated to provide an extension suffix as part of the file name. Example: rclone.conf |
| Storage System | Optional | - | Dynamic Choice Field | Remote or Local storage to execute the selected Action. Execute the Dynamic Choice Field and retrieve all the available Storages from Configuration File. Required when one of the Single storage actions is selected. |
| Source Storage | Optional | - | Dynamic Choice Field | Remote or Local source storage, where transfer Action is initiated. Execute the Dynamic Choice Field and retrieve automatically all the available Storages from Configuration File. Required when one of the Two storage actions is selected. |
| Target Storage | Optional | - | Dynamic Choice Field | Remote or Local target storage, where transferred data is stored. Execute the Dynamic Choice Field and retrieve automatically all the available Storages from Configuration File. Required when one of the Two storage actions is selected. |
| Storage Credentials | Optional | - | Credential | Credentials needed to connect to the selected Storage System. Required when one of the Single storage actions is selected. For Storage System of type local, the standard Agent's Credentials field should be provided. |
| Source Credentials | Optional | - | Credential | Credentials needed to connect to the selected Source Storage. Required when one of the Two storage actions is selected. For Storage System of type local, the standard Agent's Credentials field should be provided. |
| Target Credentials | Optional | - | Credential | Credentials needed to connect to the selected Target Storage. Required when one of the Two storage actions is selected. For Storage System of type local, the standard Agent's Credentials field should be provided. |
| Filepath | Optional | - | Text | File or directory path from where to retrieve data for the selected Action. Required when one of the Single storage actions is selected. |
| Source Filepath | Optional | - | Text | File or directory path from where to retrieve data for the selected Action. Required when one of the Two storage actions is selected. |
| Target Filepath | Optional | - | Text | File or directory path where transferred data to will be stored for the selected Action. Required when one of the Two storage actions is selected. |
| Update Credentials | Optional | False | Boolean | This field needs to be used when Rclone executable updates Fields on its Configuration File during execution, that need to be stored as Credential Fields on the Universal Controller. For example: when OneDrive Storage type is used. In this case, the existing configured token is exchanged by Rclone with a new one to establish the connection. This new token should be stored on Universal Controller for the next task execution. Please read Rclone limitations on OneDrive token refresh here. This field is used in conjunction with fields Controller URL, Controller Credentials, Refresh Storage Credentials. |
| Controller URL | Optional | - | Text | Controller URL where Storage Credentials, Source Credentials, Target Credentials are stored. Required when Update Credentials is checked. |
| Controller Credentials | Optional | - | Credentials | Controller user's credentials, used for logging and updating Storage Credentials, Source Credentials, Target Credentials will be updated. The Credentials definition should be as follows:
Required when Update Credentials is checked. |
| Refresh Storage Credentials | Optional | - | Array | This Array field is available when Update Credentials is set to True and used as a mapping table that indicates which field from the Configuration File of Rclone is used to update which Credential entry field on Universal Controller. Populate this array with the following format:
Note: Refresh of Storage/Source/Target Credentials is performed via Controller's REST API. See Network and Connectivity Requirements |
| Use Filter | Optional | -- None -- | Choice | The filter type that is applied on the Action. Available options:
Optional for all, but list type Actions. |
| Filter | Optional | - | Text | Filter as regular expression that is applied on the Action. Required when Use Filter is checked. |
| Overwrite Options | Optional | Do Not Overwrite | Choice | Options for overwriting files in the Target Storage. Available options:
Optional when one of the Two storage actions is selected. |
| Recursion Depth | Optional | 1 | Integer | Recursion depth that is applied. Default value 1 means that no recursion will be applied. Note: Use this option with caution when one of the following Actions are selected, as this might result in doing actions on unnecessary files/cloud objects. For more information on Recursion Depth you can refer to the official Rclone documentation:
|
| Error On No File Transfer | Optional | False | Boolean | When enabled and when no files are transferred, a failure exit code with value equal to 21 is raised. Optional when one of the Two storage actions is selected. |
| Dry-Run | Optional | False | Boolean | When enabled, performs a trial run with no permanent changes of the selected Action. Note: It is recommended to execute a Dry-Run as a test prior to the final task configuration when the following Actions are used:
|
| Additional Options | Optional | - | Text | Space separated Rclone options that applied to the selected Action. |
| Log Format | Optional | Text | Choice | Option for STDERR logging format. Available options:
|
...
| Attribute | Type | Description |
|---|---|---|
| cmd | string | The Rclone generated and executed command based on the input fields. |
| rc | number | The Rclone native return code. More information about Rclone exit codes can be found here. |
An example of the Extension Output for a "Copy Objects" Action is presented below.
...
| Document Link | Description |
|---|---|
| Universal Templates | User documentation for creating Universal Templates in the Universal Controller user interface. |
| Universal Tasks | User documentation for creating Universal Tasks in the Universal Controller user interface. |
| Rclone | Rclone official documentation. |
| Rclone Install | Installation instructions. |
| Rclone Downloads | Download links for Linux, Windows, MacOS. |
| Rclone Storage Systems configuration | Configuration file details. |
| Rclone Exit Codes | Rclone exit codes list. |
Changelog
ue-cloud-dt-2.0.1 (2023-07-28)
Fixes
Fixed: Retrieving STDOUT/STDERR might block a Task Instance in Running status (#33732).
ue-cloud-dt-2.0.0 (2022-10-26)
...