Panel | |
---|---|
|
Disclaimer
...
Warning |
---|
This is a major release and introduces breaking changes that might affect some users depending on their setup. Administrators are strongly advised to refer to Cloud Data Transfer for more information on the changes introduced in this release. |
Overview
This integration provides the capability to perform data transfers between cloud-based storage services and local or distributed file systems. It also provides data storage management capabilities like listing, creating, or deleting data storage objects.
Key Features
This integration is equipped with the following key features.
...
Name | Type | Description | Version Information | ||||||
---|---|---|---|---|---|---|---|---|---|
Action | Choice | Action to be performed. Available actions:
| Introduced in 1.0.0 | ||||||
Configuration File | Script | Rclone Configuration file, which contains all required parameters to connect to the Storage System, Source Storage and Target Storage. The filename should have the ".conf" suffix. Example: rclone.conf | Introduced in 1.0.0 | ||||||
Storage System | Dynamic Choice Field | Remote or Local storage to execute the selected Action. Execute the Dynamic Choice Field and retrieve all the available Storages from Configuration File. Required when one of the Single storage actions is selected. | Introduced in 1.0.0 | ||||||
Source Storage | Dynamic Choice Field | Remote or Local source storage, where transfer Action is initiated. Execute the Dynamic Choice Field and retrieve automatically all the available Storages from Configuration File. Required when one of the Two storage actions is selected. | Introduced in 1.0.0 | ||||||
Target Storage | Dynamic Choice Field | Remote or Local target storage, where transferred data is stored. Execute the Dynamic Choice Field and retrieve automatically all the available Storages from Configuration File. Required when one of the Two storage actions is selected. | Introduced in 1.0.0 | ||||||
Storage Credentials | Credential | Credentials needed to connect to the selected Storage System. Required when one of the Single storage actions is selected. For Storage System of type local , the standard Agent's Credentials field should be provided. | Introduced in 1.0.0 | ||||||
Source Credentials | Credential | Credentials needed to connect to the selected Source Storage. Required when one of the Two storage actions is selected. For Storage System of type local , the standard Agent's Credentials field should be provided. | Introduced in 1.0.0 | ||||||
Target Credentials | Credential | Credentials needed to connect to the selected Target Storage. Required when one of the Two storage actions is selected. For Storage System of type local , the standard Agent's Credentials field should be provided. | Introduced in 1.0.0 | ||||||
Filepath | Text | File or directory path from where to retrieve data for the selected Action. Required when one of the Single storage actions is selected. | Introduced in 1.0.0 | ||||||
Source Filepath | Text | File or directory path from where to retrieve data for the selected Action. Required when one of the Two storage actions is selected. | Introduced in 1.0.0 | ||||||
Target Filepath | Text | File or directory path where transferred data will be stored for the selected Action. Required when one of the Two storage actions is selected. | Introduced in 1.0.0 | ||||||
Update Credentials | Checkbox | If the remote storage uses OAuth for its authorization, Rclone can be adjusted to refresh the respective token in its Configuration File during execution time. Enabling this field (and as soon as Rclone is adjusted to refresh tokens), will ensure that the runtime-updated information from Rclone, will get provisioned to the respective Credential Fields on the Universal Controller. The default setting is unchecked. | Introduced in 1.0.0 | ||||||
Controller URL | Text | Controller URL where Storage Credentials, Source Credentials, Target Credentials are stored. Required when Update Credentials is checked. | Introduced in 1.0.0 | ||||||
Controller Credentials | Credentials | Controller user's credentials, used for logging and updating Storage Credentials, Source Credentials, Target Credentials. The Credentials definition should be as follows:
Required when Update Credentials is checked. | Introduced in 1.0.0 | ||||||
Refresh Storage Credentials
| Array | This Array field is available when Update Credentials is checked. Is used as a mapping table between the Configuration File locator that has been updated by Rclone during execution, and its corresponding Universal Controller Credential field. Populate this array with the following format:
The result of this action is that the Configuration File's locator value, is copied to the provided Storage Credential's field. Each line of this Array is evaluated, and related mapping is performed. Note: Refresh of Storage/Source/Target Credentials is performed via Controller's REST API. See Network Cloud Data Transfer Requirements. | Introduced in 1.0.0 | ||||||
Use Filter | Choice | The filter type that is applied on the Action. Available options:
Optional for all, but Create Object Action. | Introduced in 1.0.0 | ||||||
Filter | Text | Filter as regular expression that is applied on the Action. Required when Use Filter is checked. | Introduced in 1.0.0 | ||||||
Overwrite Options | Choice | Options for overwriting files in the Target Storage. Available options:
Optional when one of the Two storage actions is selected. | Introduced in 1.0.0 | ||||||
Recursion Depth | Integer | Recursion depth that is applied. Default value 1 means that no recursion will be applied. Note: Use this option with caution when one of the following Actions are selected, as this might result in doing actions on unnecessary files/cloud objects. For more information on Recursion Depth, you can refer to the official Rclone documentation:
The default value is 1. | Introduced in 1.0.0 | ||||||
Error On No File Transfer | Checkbox | When enabled and when no files are transferred a failure exit code with value equal to 21 is raised. The default setting is unchecked. | Introduced in 1.0.0 | ||||||
Dry-Run | Checkbox | When enabled, performs a trial run with no permanent changes of the selected Action. Note: It is recommended to execute a Dry-Run as a test prior to the final task configuration when the following Actions are used:
The default value is unchecked. | Introduced in 1.0.0 | ||||||
Additional Options | Text | Space-separated Rclone options applied to the selected Action. | Introduced in 1.0.0 | ||||||
Log Format | Choice | Option for STDERR logging format. Available options:
| Introduced in 1.0.0 |
...
In the Variable Tab of the Cloud Data Transfer configuration, create a new Variable. Its value should be the name of an existing Resolvable Credential:
Example of a Configuration File for AWS S3 Storage, using a Credential Function with a variable that contains the Credential Name
...
An example of such a configuration can be found below:
Note |
---|
Please read the Rclone limitations on OneDrive token refresh. |
...