...
| Template Name | Extension Name | Extension Version |
|---|---|---|
| Cloud Data Transfer | ue-cloud-dt | 23.0.1 |
...
| 0 |
| Warning |
|---|
This is a major release and introduces breaking changes that might affect some customers depending on their setup. Administrators are strongly advised to refer to Cloud Data Transfer for more information on the changes introduced in this release. |
Software Requirements
This integration requires a Universal Agent, a Python runtime to execute the Universal Task, and a Rclone executable.
Software Requirements for Universal Template and Universal Task
Requires Python 3.7 .0 or higher. Tested with the Universal Agent bundled Python distribution.
...
- Universal Agent for Windows x64 Version 7.2.0.0 and later with python Python options installed.
- Universal Agent for Linux Version 7.2.0.0 and later with python Python options installed.
Software Requirements for Universal Controller
...
| Note |
|---|
| This Universal Extension has been tested with Rclone v1.58.1. It should be working with later Rclone versions, as long as these are backwards backward compatible. |
Network and Connectivity Requirements
Universal Agent should be able to establish a connection with:
- The remote storages that are provided as input for the Universal Task.
- The Universal Controller host. This is required only in case Refresh Storage Credentials field is used.
...
This Universal Extension is an interface for Rclone and as an interface, it supports the following key features:
- Actions
- Copy, move, and synchronize data between two storages.
- Copy a URL's content and to cloud or local destination without saving it in temporary storage.
- List data on a storage, including listing with details or in JSON format for machine parsing.
- Create objects on a storage.
- Delete objects from a storage.
- Features:
- Fast transfers for objects stored in the same region.
- Preserves always timestamps and verifies checksums.
- Supports encryption, caching, compression, and chunking.
- Dynamic token updates for OneDrive Business cloud storage, observing the OneDrive business refresh token flow.
- Support for dry runs. Allows users to execute a Universal Task without making any permanent changes on to the target storage.
- Advanced filtering capability for files or objects to be listed or transferred.
- Option to mark the Universal Task as Failed when no files have been transferred.
- List of overwrite options for existing data.
- Additional customized options.
- Rich Observabilty capabilities.
- Output
- Progress of the selected Action is visible, during Universal Task Instance execution.
- Text or JSON formatted output.
...
| Note |
|---|
This integration should work properly against other Storage Systems, as long as the integration interface (see chapter Configure Universal Task) and functionalities listed in this document fulfil fulfill the needs of the Storage System. For functionalities required for specific Storage Systems, users and customers are encouraged to open a Feature Request in our Customer Support Portal. |
...
This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.
To import the Universal Template into your Controller, follow the instructions here.
When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list.
...
Setup Rclone Configuration File
The configuration file is following follows the INI format , and contains all required parameters and credentials to connect to a Storage System. All configuration options per Storage System can be found in the respective "Config Rclone documentation" here.
It is advised that account credentials, tokens, or any other essential information to be passed as an encrypted Resolvable Credential field. To set a value with the respective Credential field, please make sure to use the appropriate Credential function inside the configuration file.
Note 'Runtime Password' Credentials field supports values with up to 512 characters.
Hard-coded values can be provided in the script file, as well. However, it is not advised for security reasons.
It is recommended to configure a storage system section, one to one with the Credential fields in this Universal Extension (please refer to the Input Fields Cloud Data Transfer section below).
For local file system Storages, no credentials are required in the configuration file. The selected action will be executed with the Credentials that are provided in the "Agent Details" of the Universal Task. If no Credentials are provided for the used Agent, then the same user that runs ubroker daemon will be used.
Script Type should be Data, and option Resolve UAC Variables should be enabled.
The indicative Configuration File below , contains the basic connection parameters (flags):
...
| Field | Input type | Default value | Type | Description |
|---|---|---|---|---|
| Action | Required | Copy Objects | Choice | Action to be performed. Available actions:
|
| Configuration File | Required | - | Script | Rclone Configuration file, that which contains all required parameters to connect to the Storage System, Source Storage and Target Storage. It is indicated to provide an extension suffix as part of the file name. The filename should have the ".conf" suffix. Example: rclone.conf conf |
| Storage System | Optional | - | Dynamic Choice Field | Remote or Local storage to execute the selected Action. Execute the Dynamic Choice Field and retrieve all the available Storages from Configuration File. Required when one of the Single storage actions is selected. |
| Source Storage | Optional | - | Dynamic Choice Field | Remote or Local source storage, where transfer Action is initiated. Execute the Dynamic Choice Field and retrieve automatically all the available Storages from Configuration File. Required when one of the Two storage actions is selected. |
| Target Storage | Optional | - | Dynamic Choice Field | Remote or Local target storage, where transferred data is stored. Execute the Dynamic Choice Field and retrieve automatically all the available Storages from Configuration File. Required when one of the Two storage actions is selected. |
| Storage Credentials | Optional | - | Credential | Credentials needed to connect to the selected Storage System. Required when one of the Single storage actions is selected. For Storage System of type local, the standard Agent's Credentials field should be provided. |
| Source Credentials | Optional | - | Credential | Credentials needed to connect to the selected Source Storage. Required when one of the Two storage actions is selected. For Storage System of type local, the standard Agent's Credentials field should be provided. |
| Target Credentials | Optional | - | Credential | Credentials needed to connect to the selected Target Storage. Required when one of the Two storage actions is selected. For Storage System of type local, the standard Agent's Credentials field should be provided. |
| Filepath | Optional | - | Text | File or directory path from where to retrieve data for the selected Action. Required when one of the Single storage actions is selected. |
| Source Filepath | Optional | - | Text | File or directory path from where to retrieve data for the selected Action. Required when one of the Two storage actions is selected. |
| Target Filepath | Optional | - | Text | File or directory path where transferred data to will be stored for the selected Action. Required when one of the Two storage actions is selected. |
| Update Credentials | Optional | False | Boolean | This field needs to be used when the Rclone executable updates Fields fields on its Configuration File during execution, that which need to be stored as Credential Fields on the Universal Controller. For example: when OneDrive Storage type is used. In this case, the existing configured token is exchanged by Rclone with a new one to establish the connection. This new token should be stored on Universal Controller for the next task execution. Please read Rclone limitations on OneDrive token refresh here. This field is used in conjunction with fields Controller URL, Controller Credentials, Refresh Storage Credentials. |
| Controller URL | Optional | - | Text | Controller URL where Storage Credentials, Source Credentials, Target Credentials are stored. Required when Update Credentials is checked. |
| Controller Credentials | Optional | - | Credentials | Controller user's credentials, used for logging and updating Storage Credentials, Source Credentials, Target Credentials will be updated. The Credentials definition should be as follows:
Required when Update Credentials is checked. |
| Refresh Storage Credentials | Optional | - | Array | This Array field is available when Update Credentials is set to True and used as a mapping table that indicates which field from the Configuration File of Rclone is used to update which Credential entry field on Universal Controller. Populate this array with the following format:
Note: Refresh of Storage/Source/Target Credentials is performed via Controller's REST API. See Network and Connectivity Requirements |
| Use Filter | Optional | -- None -- | Choice | The filter type that is applied on the Action. Available options:
Optional for all, but list type Actions Create Object Action. |
| Filter | Optional | - | Text | Filter as regular expression that is applied on the Action. Required when Use Filter is checked. |
| Overwrite Options | Optional | Do Not Overwrite | Choice | Options for overwriting files in the Target Storage. Available options:
Optional when one of the Two storage actions is selected. |
| Recursion Depth | Optional | 1 | Integer | Recursion depth that is applied. Default value 1 means that no recursion will be applied. Note: Use this option with caution when one of the following Actions are selected, as this might result in doing actions on unnecessary files/cloud objects. For more information on Recursion Depth, you can refer to the official Rclone documentation:
|
| Error On No File Transfer | Optional | False | Boolean | When enabled and when no files are transferred , a failure exit code with value equal to 21 is raised. Optional when one of the Two storage actions is selected. |
| Dry-Run | Optional | False | Boolean | When enabled, performs a trial run with no permanent changes of the selected Action. Note: It is recommended to execute a Dry-Run as a test prior to the final task configuration when the following Actions are used:
|
| Additional Options | Optional | - | Text | Space-separated Rclone options that applied to the selected Action. |
| Log Format | Optional | Text | Choice | Option for STDERR logging format. Available options:
|
Task Examples
Configuration File Script
...
| Note |
|---|
In the following example, the Credentials field "Token" used in "one_drive_target" of "token" field inside Configuration File will be updated with the refreshed token. |
...
List Cloud Storage
Synchronize local file system to Cloud Storage
...
The exit codes for this Universal Extension are described below.
| Exit | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Code | Status | ||||||||
| Status Description | |||||||||
| 0 | SUCCESS | Successful Execution | SUCCESS: Successful Task executionSuccess | Task executed successfully. | |||||
| 1 | FAILFailed | Execution Failed | Execution
| 21 | FAIL | Failed Execution | FAIL: "<Error Description>" | ||
| 20 | Failed | Data Validation Error: "<Error Description>" | |||||||
| 21 | Failed | Execution Failed: No Files were transferred. | |||||||
| 22 | FAILFailed | Execution Failed | ExecutionFAIL: Cloud Data Transfer task: Task executed successfully, but updating Credentials failed. | ||||||
Extension Output
In the context of a workflow, subsequent tasks can rely on the information provided by this integration as Extension Output.
...
| Case Scenario | Result |
|---|---|
| Action is other than List, executed successfully, and is not Dry-Run. | true |
| Action executed successfully, and is Dry-Run. | false |
| Action returned exit code 21. | false |
| Αction executed with failure. | false |
The Extension output contains the attribute result. Attribute result contains the following sub-attributes:
| Attribute | Type | Description |
|---|---|---|
| cmd | string | The Rclone generated and executed command command that was generated based on the input fields and executed by Rclone. |
| rc | number | The Rclone native return code. More information about Rclone exit codes can be found here. |
An example of the Extension Output for a "Copy Objects" Action is presented below.
| No Format |
|---|
{ "exit_code": 0, "status_description": " |
...
Task |
...
executed successfully", " |
...
invocation": { |
...
"extension": "ue-cloud-dt", "version": " |
...
3.0.0", "fields": { "action": "Copy Objects", " |
...
config": "rclone.conf", "log_level": "DEBUG", " |
...
log_ |
...
format": |
...
"JSON", "storage_ |
...
system": null, " |
...
storage_ |
...
type": " |
...
local", " |
...
storage_credentials |
...
": |
...
null, " |
...
source_storage": " |
...
ue_aws_ |
...
s3_ |
...
source", " |
...
source_credentials |
...
": |
...
{ " |
...
user": |
...
"ue-aws-s3", |
...
"password": " |
...
****", |
...
"token": |
...
null, " |
...
key_ |
...
location": |
...
null, |
...
"passphrase": null }, " |
...
target_storage": " |
...
one_drive |
...
_target", "target_credentials": { |
...
...
"user": "rclone_one_drive", |
...
|
...
...
"password": "****", |
...
|
...
...
" |
...
token": |
...
null, " |
...
key_ |
...
location": |
...
null, " |
...
passphrase": |
...
null |
...
}, " |
...
controller_ |
...
credentials": |
...
null, " |
...
filepath": |
...
null, " |
...
source_filepath": |
...
"john-bucket", " |
...
target_ |
...
filepath": |
...
"john-drive", " |
...
use_ |
...
filter": " |
...
Include", |
...
"filter": "*.py", |
...
...
...
" |
...
additional_options": |
...
null, " |
...
overwrite_options": " |
...
Do Not Overwrite", "max_depth": 1, "error_on_no_transfer": false, "dry_run": true, " |
...
update_credentials": |
...
false, |
...
STDOUT and STDERR
STDOUT and STDERR populated content can be changed in future versions of this extension without notice. Backward compatibility is not guaranteed.
STDOUT/STDERR of Rclone is redirected to Task STDOUT/STDERR respectively and propagated during task execution. This is particularly useful for long-lasting data transfers. Retrieve Output while Task Instance is Running and press Refresh to get the latest STDOUT/STDERR.
Integration Modifications
Modifications applied by users or customers, before or after import, might affect the supportability of this integration. The following modifications are discouraged to retain the support level as applied for this integration.
- Python code modifications should not be done.
- Template Modifications
- General Section
- "Name", "Extension", "Variable Prefix", "Icon" should not be changed.
- Universal Template Details Section
- "Template Type", "Agent Type", "Send Extension Variables", "Always Cancel on Force Finish" should not be changed.
- Result Processing Defaults Section
- Success and Failure Exit codes should not be changed.
- Success and Failure Output processing should not be changed.
- Fields Restriction Section
- Default configured values should not be changed.
- General Section
Users and customers are encouraged to report defects, or feature requests at Stonebranch Support Desk.
Document References
This document references the following documents.
...
Changelog
"controller_url": null,
"refresh_storage_credentials": null,
"storage_credentials_name": null,
"source_credentials_name": "aws_se",
"target_credentials_name": "one_drive",
"controller_credentials_name": null,
"configuration_file_name": "rclone.conf"
}
},
"changed": true,
"result": {
"cmd": "rclone --config=//var/opt/universal/tmp/b4a333c4-497e-425c-8ce3-f8c5bdad9150.con --log-level=DEBUG --use-json-log --max-depth=1 --dry-run --progress --no-traverse --ignore-existing copy ue_aws_s3_source:john-bucket one_drive_target:john-drive --include *.py",
"rc": 0
}
} |
STDOUT and STDERR
| Note |
|---|
STDOUT and STDERR provide additional information to the User. The populated content can be changed in future versions of this extension without notice. Backward compatibility is not guaranteed. |
STDOUT/STDERR of Rclone is redirected to Task STDOUT/STDERR respectively and propagated during task execution. This is particularly useful for long-lasting data transfers. Retrieve Output while Task Instance is Running and press Refresh to get the latest STDOUT/STDERR.
Observability
Metrics are generated and exported by this integration, contributing to the UAC data observability, delivered since Universal Automation Center 7.5.0.0.
Metric: ue.cdt.rclone.speed
Name | Instrument Type | Unit (UCUM) | Attributes | Instrumented By | Description |
|---|---|---|---|---|---|
| Histogram | By/s | As defined on Metric Attributes. | Universal Controller (Through Universal Event Templates) | The average speed at which rclone transfers files. |
| Note |
|---|
When the transfer time is too small, the speed becomes too big (Bytes/Sec) and is displayed as 0 by RClone. In such cases, the transfer speed statistic is ignored and no relevant metric is sent. |
Metric: ue.cdt.rclone.duration
Name | Instrument Type | Unit (UCUM) | Attributes | Instrumented By | Description |
|---|---|---|---|---|---|
| Histogram | s | As defined on Metric Attributes. | Universal Controller (Through Universal Event Templates) | The time took to complete the entire file transfer process, from initiation to completion. |
Metric: ue.cdt.rclone.transfers
Name | Instrument Type | Unit (UCUM) | Attributes | Instrumented By | Description |
|---|---|---|---|---|---|
| Histogram | {transfers} | As defined on Metric Attributes. | Universal Controller (Through Universal Event Templates) | The total number of file transfers. |
Metric Attributes
The following attributes are applicable for ue.cdt.rclone.speed, ue.cdt.rclone.duration and ue.cdt.rclone.transfers.
Attribute Name | Enabled | Description |
|---|---|---|
action | By Default | The selected action of the Universal Task. |
source | By Default | The source storage of the files. |
target | By Default | The source file path. |
Other Observability Configuration Options
Administrators are able to update the Universal Event Template to control additional attributes thatcan be sent out for Universal Event-based Metrics. However, this configuration should be approached with caution. When the possible distinct values of those attributes are high, that might lead to high cardinality issues.
Administrators can activate them with caution as follows:
Go to Administration → Universal Templates
Select the Universal Template
Click “Event Templates” Tab
Select the Name of the Event Template you want to update, which represents the metric you are interested in.
Update the “Optional Metric Labels” List the required Metric Labels.
How To
Import Grafana Dashboard
Users can benefit from a ready-to-use sample dashboard that this downloadable integration offers when observability features are used. It is located under the /observability/grafana/ directory inside the downloadable zip file from Stonebranch Integration Hub. Administrators should refer to the official Grafana User Guide on how to import a Grafana Dashboard.
| Info |
|---|
Dashboard’s Prometheus data source is configured as a variable and thus needs to be mapped to an existing Data Source configured on the target Grafana instance. |
Integration Modifications
Modifications applied by users or customers, before or after import, might affect the supportability of this integration. The following modifications are discouraged to retain the support level as applied for this integration.
Python code modifications should not be done.
Template Modifications
General Section
"Name", "Extension", "Variable Prefix", and "Icon" should not be changed.
Universal Template Details Section
"Template Type", "Agent Type", "Send Extension Variables", and "Always Cancel on Force Finish" should not be changed.
Result Processing Defaults Section
Success and Failure Exit codes should not be changed.
Success and Failure Output processing should not be changed.
Fields Restriction Section
The setup of the template does not impose any restrictions, However with respect to the "Exit Code Processing Fields" section.Success/Failure exit codes need to be respected.
In principle, as STDERR and STDOUT outputs can change in follow-up releases of this integration, they should not be considered as a reliable source for determining the success or failure of a task.
Event Template configuration related to “Metric Label Attributes” & “Optional Metric Labels” is allowed. However, administrators should be cautious of high cardinality scenarios that might occur
Users and customers are encouraged to report defects, or feature requests at Stonebranch Support Desk.
Document References
This document references the following documents.
| Document Link | Description |
|---|---|
| Universal Templates | User documentation for creating Universal Templates in the Universal Controller user interface. |
| Universal Tasks | User documentation for creating Universal Tasks in the Universal Controller user interface. |
| Rclone | Rclone official documentation. |
| Rclone Install | Installation instructions. |
| Rclone Downloads | Download links for Linux, Windows, and MacOS. |
| Rclone Storage Systems configuration | Configuration file details. |
| Rclone Exit Codes | Rclone exit codes list. |
Changelog
ue-cloud-dt-3.0.0 (2023-10-09)
Enhancements
Changed: The progress bar, available on Universal Controllers of version 7.3 and above, now syncs with RClone's transfer completion (#34332).
- Changed: Improved clarity of RClone's log messages with respect to log level (#34332).
Added: Introduced Observability metrics enabling enhanced visibility and understanding of behavior for Task Instances. This feature is enabled on Universal Controllers of version 7.5 and above (#34330).
- Added: Filter options are available for all list actions (#34455).
Deprecations and Breaking Changes
Breaking Change: Updated the status descriptions to be more informative (#34335).
Warning Tasks or workflows evaluating the "Status Description" of the task Instance, either programmatically or within UAC, might be affected by it. In that case, they need to conform to the new "Status Description" Text
Breaking Change: Extension Invocation Fields > Some fields are renamed for better readability (#34335).
Breaking Change: Extension Invocation Fields > Credential fields are displayed as an object for better readability (#34335).
Fixes
- Fixed: RClone's command will no longer include duplicate flags (#34265).
Other
- Extra: The extension is supplemented with a Grafana dashboard that, by utilizing the produced observability metrics, calculates and displays statistical data (#34338).
ue-cloud-dt-2.0.1 (2023-07-28)
Fixes
- Fixed: Retrieving STDOUT/STDERR might block a Task Instance in Running status (#33732).
ue-cloud-dt-2.0.0 (2022-10-26)
Deprecations and Breaking Changes
- Breaking Change: Universal Template Name rename.
This version of Universal Extension
...
- will have no effect on existing tasks created with Universal Template Cloud Data Transfer and extension archive ue-cloud-ft-1.0.0. Future updates on this integration will take place on top of ue-cloud-dt-2.0.0.
ue-cloud-ft-1.0.0 (2022-09-02)
Enhancements
- Added: Initial version (#28787)