Inter-Cloud Data Transfer

Disclaimer

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at Stonebranch Integration Hub - Terms of Use.


Version Information

Template Name

Extension Name

Version

Status

Cloud Data Transfer

ue-cloud-dt

4 (Current 4.0.2)

Fixes and new features are introduced.

Cloud Data Transferue-cloud-dt3Hot Fixes Only (Until UAC 7.3 is End of Support)
Cloud Data Transferue-cloud-dt1 & 2 End of Support

Refer to Changelog for version history information.

Version 4.0.0 is a major release and introduces breaking changes that might affect some users depending on their setup. Administrators are strongly advised to refer to Changelog for more information on the changes introduced in this release. Refer also to Migration Guide to apply changes in existing task definitions.  

Overview

This integration provides the capability to perform data transfers between cloud-based storage services and local or distributed file systems. It also provides data storage management capabilities like listing, creating, or deleting data storage objects. 


Key Features

This integration is equipped with the following key features.


FeatureDescription
Data management and synchronization
  • Copy, move, and synchronize data between two storages.
  • Copy a URL's content to a cloud or local destination without saving it in temporary storage.
  • List data on a storage service, including listing with details or in JSON format for machine parsing.
  • Create objects on a storage.
  • Delete objects from a storage.
Efficient and secure data transfers
  • Fast transfers for objects stored in the same region.
  • Preserve timestamps and checksums verification.
  • Transfers can optionally support encryption, caching, compression, and chunking.
  • Dynamic token refresh for OneDrive cloud storage, observing the OAuth2 authorization and sign-in flow.
  • Support for dry runs. Allows users to execute a Universal Task without making any permanent changes to the target storage.
  • Advanced filtering capability for files or objects to be listed or transferred.
Complementary capabilities
  • Additional customized options.
  • Option to mark the Universal Task as Failed when no files have been transferred.
  • Progress of the selected Action is visible, during Universal Task Instance execution.
  • Text or JSON formatted output.
Observability
  • Rich observability capabilities (starting from UAC 7.5.0.0)


Requirements

This integration requires a Universal Agent, a Python runtime to execute the Universal Task.

Area

Details

Python Version

Requires Python 3.7 or 3.11.  Tested with Python 3.7.6 and Python 3.11.6.

Universal Agent Compatibility

  • Compatible with Universal Agent for Windows x64 and version >= 7.4.0.0.

  • Compatible with Universal Agent for Linux and version >= 7.4.0.0.

Universal Controller Compatibility

Universal Controller Version >= 7.4.0.0.

Network and Connectivity

Universal Agent should be able to establish a connection with:

  • The remote storage providers used at the Universal Task Definition.
  • The Universal Controller host. This is required only in case Refresh Storage Credentials field is used



Supported Actions

This integration supports multiple actions which can be grouped into the following categories: 

CategoryActionsRequired Input Fields
Single storage actions
  • List Objects
  • List Directories
  • List Objects With Details
  • List Objects In JSON Format
  • Create Object
  • Delete Object
  • Delete Empty Directory
  • Storage System
  • Storage Credentials
  • Filepath
Two storage actions
  • Copy Objects
  • Copy And Rename
  • Copy From URL
  • Move Objects
  • Move And Rename
  • Synchronize
  • Source Storage
  • Source Credentials
  • Source Filepath
  • Target Storage
  • Target Credentials
  • Target Filepath

Action Type: Single Storage Action

Single storage actions involve listing and managing existing data objects in a local or remote storage. 


Configuration Examples


User Scenario:

List Directories action, using a filter and with output log in Text format.

User Scenario:

Create Object action and with output as JSON format.


Action Output


Output Type

Description

Examples

EXTENSION

The extension output provides the following information:

  • exit_code,  status_description: General info regarding the task execution. For more information, users can refer to the Exit Codes section.

  • invocation.fields: The task configuration used for this task execution.

  • result.cmd: The executed command line.

  • result.rc: The return code from the executed command line.

Successful Execution
{
	"exit_code": 0,
	"status_description": "Task executed successfully",
	"invocation": {
		"extension": "ue-cloud-dt",
		"version": "4.0.0",
		"fields": {
			"action": "List Directories",
             ...
		}
	},
	"changed": false,
	"result": {
		"cmd": "rclone lsd ue_aws_s3:john-drive/--config=C:\\DATA\\tmp\\54cd05b2-1528-46d2-8f9c- 8bc09dfdbf4.conf --log-level=DEBUG --max-depth=4 --use-json-log --exclude ue_execution/compiled",
		"rc": 0
	}
}
Failed Execution
{
    "exit_code": 1,
    "status_description": "Execution Failed",
    "invocation": {
        "extension": "ue-cloud-dt",
        "version": "4.0.0",
        "fields": {
             "action": "List Directories",
              ...
         },
    "changed": false,
    "result": {
        "cmd": "rclone ls ue_aws_s3:john-drive/
--config=/home//ee1152e6-2afe-4383-b611-dfe4eb3c7fde.con 
--log-level=DEBUG --max-depth=1 --use-json-log",
        "rc": 1
    }
}

STDOUT

The executed command's output.

STDOUT Example > List Directories
-1 2023-08-30 13:24:22 -1 analyses
-1 2023-08-11 15:31:44 -1 logs
-1 2023-07-10 13:59:55 -1 macros
-1 2023-07-10 13:59:55 -1 models
-1 2023-07-10 13:59:55 -1 seeds
-1 2023-07-10 13:59:55 -1 snapshots
-1 2023-07-10 13:59:55 -1 tests
-1 2023-07-10 13:59:55 -1 ue_execution


Action Type: Two Storage Action

Two storage actions involve data transfers between local or cloud storages.  

Configuration Examples



User Scenario:

Copy files matching filenames with pattern “2024_March*.csv”, from local storage of Universal Agent to AWS S3. Exclude files matching “2024_March*.txt”.

User Scenario:

Synchronization of source and target cloud storages as a dry run execution. In the scenario that no files are to be synced the task instance status is marked as “Failed".  


Action Output


Output Type

DescriptionExamples
EXTENSION

The extension output provides the following information:

  • exit_code,  status_description: General info regarding the task execution. For more information users can refer to the Exit Codes section.

  • invocation.fields: The task configuration used for this task execution.

  • result.cmd: The executed command line.

  • result.rc: The return code from the executed command line.

Successful Execution
{
     "exit_code": 0,
     "status_description": "Task executed successfully",
     "invocation": {
          "extension": "ue-cloud-dt",
          "version": "4.0.0",
          "fields": {
              "action": "Copy Objects",
              ...
          }
      },
      "changed": true,
      "result": {
          "cmd": "rclone copy local_ua:C:\\Users\\test\\Desktop\\dbt one_drive_target:john-drive --config=C:\\DATA\\tmp\\74f533e1-07c5-46f6-a48e-698aa55c7dd7.conf --log-level=DEBUG --max-depth=1 --progress --no-traverse --ignore-times --ignore-size --use-json-log --s3-no-check-bucket"
          "rc": 0
      }
}  
Failed Execution
{
    "exit_code": 1,
    "status_description": "Execution Failed",
     "invocation": {
          "extension": "ue-cloud-dt",
          "version": "4.0.0",
          "fields": {
              "action": "Copy Objects",
              ...
          }
      },
      "changed": true,
      "result": {
          "cmd": "rclone copy local_ua:C:\\Users\\test\\Desktop\\dbt one_drive_target:john-drive --config=C:\\DATA\\tmp\\74f533e1-07c5-46f6-a48e-698aa55c7dd7.conf --log-level=DEBUG --max-depth=1 --progress --no-traverse --ignore-times --ignore-size --use-json-log --s3-no-check-bucket",
          "rc": 1
      }
}  
STDOUTThe executed command's output.
STDOUT Example > Copy Object
Transferred:                          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:                       1.3sTransferred: 29 B / 300.002 MiB, 0%, 0 B/s, ETA -
Transferred:                          0 / 7, 0%
Elapsed time:                          1.8s
Transferring:
*                                   .gitignore:100% /29, 0/s, -
*                                   100MB.bin: 0% /100Mi, 0/s, -
*                                   100MB2.bin: 0% /100Mi, 0/s, -
*                                   100MB3.bin: 0% /100Mi, 0/s, -Transferred: 3.489 MiB 
/ 300.002 MiB, 1%, 0 B/s, ETA -
Transferred:                        1 / 7, 14%
Elapsed time:                      2.3s
Transferring:
*                                  100MB.bin: 1% /100Mi, 0/s, -
*                                  100MB2.bin: 0% /100Mi, 0/s, -
*                                  100MB3.bin: 0% /100Mi, 0/s, -
*                                  README.md:100% /571, 0/s, -Transferred: 18.834 MiB 
/ 300.002 MiB, 6%, 4.572 MiB/s,
ETA 1m1s
Transferred:                    2 / 7, 29%
Elapsed time:                   2.8s

Input Fields 


NameTypeDescriptionVersion Information
ActionChoiceAction to be performed.

Available actions:
  • Copy Objects (default)
  • Copy And Rename
  • Copy From URL
  • Move Objects
  • Move And Rename
  • List Objects
  • List Directories
  • List Objects With Details
  • List Objects In JSON Format
  • Create Object
  • Delete Object
  • Delete Empty Directory
  • Synchronize
Introduced in 1.0.0
Configuration FileScript

This integration depends on Rclone. This field models the Rclone Configuration file, which contains all required parameters to connect to the Storage System, Source Storage and Target Storage. The filename should have the ".conf" suffix.

Example: rclone.conf

Introduced in 1.0.0
Storage SystemDynamic Choice FieldRemote or Local storage to execute the selected Action.
Execute the Dynamic Choice Field and retrieve all the available Storages from Configuration File.

Required when one of the Single storage actions is selected.
Introduced in 1.0.0
Source StorageDynamic Choice FieldRemote or Local source storage, where transfer Action is initiated.
Execute the Dynamic Choice Field and retrieve automatically all the available Storages from Configuration File.

Required when one of the Two storage actions is selected.
Introduced in 1.0.0
Target StorageDynamic Choice FieldRemote or Local target storage, where transferred data is stored.
Execute the Dynamic Choice Field and retrieve automatically all the available Storages from Configuration File.

Required when one of the Two storage actions is selected.
Introduced in 1.0.0
Storage CredentialsCredential

Credentials needed to connect to the selected Storage System.
For Storage System of type local, the standard Agent's Credentials field should be provided.

From Universal Controller 7.6 onwards a variable that holds the Credential Name can be used when related checkbox is selected. In this case the content of this field should be ${variable_name}


Required when one of the Single storage actions is selected.

Introduced in 1.0.0
Source CredentialsCredential

Credentials needed to connect to the selected Source Storage.

From Universal Controller 7.6 onwards a variable that holds the Credential Name can be used when related checkbox is selected. In this case the content of this field should be ${variable_name}

Required when one of the Two storage actions is selected. For Storage System of type local, the standard Agent's Credentials field should be provided.

Introduced in 1.0.0
Target CredentialsCredential

Credentials needed to connect to the selected Target Storage.
From Universal Controller 7.6 onwards a variable that holds the Credential Name can be used when related checkbox is selected. In this case the content of this field should be ${variable_name}


Required when one of the Two storage actions is selected.

For Storage System of type local, the standard Agent's Credentials field should be provided.

Introduced in 1.0.0
FilepathTextFile or directory path from where to retrieve data for the selected Action.

Required when one of the Single storage actions is selected.
Introduced in 1.0.0
Source FilepathTextFile or directory path from where to retrieve data for the selected Action.

Required when one of the 150079703 is selected.
Introduced in 1.0.0
Target FilepathTextFile or directory path where transferred data will be stored for the selected Action.

Required when one of the Two storage actions is selected.
Introduced in 1.0.0
Update CredentialsCheckbox 

If the remote storage uses OAuth for its authorization, Rclone can be adjusted to refresh the respective token in its Configuration File during execution time. Enabling this field, will ensure that the runtime-updated token from Rclone, will get provisioned to the respective Credential Field on the Universal Controller.

This field is used in conjunction with fields Controller URL, Controller Credentials, and optionally the Refresh Storage Credentials. See the respective examples in Refresh Storage Credentials chapter.


The default setting is unchecked.

Introduced in 1.0.0
Controller URLTextController URL where Storage Credentials, Source Credentials, Target Credentials are stored.

Required when Update Credentials is checked.
Introduced in 1.0.0
Controller CredentialsCredentialsController user's credentials, used for logging and updating Storage Credentials, Source Credentials, Target Credentials. The Credentials definition should be as follows:
  • User as "Runtime User".
  • User Password as "Runtime Password".
Note: The provided user should have sufficient Credential permissions.

Required when Update Credentials is checked.
Introduced in 1.0.0

Refresh Storage Credentials

Array

Note: This Array is not required to be configured in case the Storage type is "One Drive".

This Array field is available when Update Credentials is checked. Is used as a mapping table between the Configuration File locator that has been updated by Rclone during execution, and its corresponding Universal Controller Credential field.

Populate this array with the following format:

  • Configuration File Details: "<Storage/Source/Target Storage>"."<Configuration File's field to be refreshed by Rclone>"
  • Storage Credentials Details: "<Storage/Source/Target Credentials name>"."<Storage/Source/Target Credentials field to be updated>"

The result of this action is that the Configuration File's locator value, is copied to the provided Storage Credential's field. Each line of this Array is evaluated, and related mapping is performed. See the Refresh authorization fields for other remotes.

Note: Refresh of Storage/Source/Target Credentials is performed via Controller's REST API. See Network Requirements .

Introduced in 1.0.0
Use FilterChoiceThe filter type that is applied on the Action.

Available options:
  • -- None – (default)
  • Include
  • Exclude

Optional for all, but Create Object Action.
Introduced in 1.0.0
FilterTextFilter as regular expression that is applied on the Action.

Required when Use Filter is checked.
Introduced in 1.0.0
Overwrite OptionsChoiceOptions for overwriting files in the Target Storage.

Available options:
  • Overwrite
  • Do Not Overwrite (default)
  • Append Timestamp In Filename

Optional when one of the Two storage actions is selected.
Introduced in 1.0.0
Recursion DepthIntegerRecursion depth that is applied. Default value 1 means that no recursion will be applied.

Note: Use this option with caution when one of the following Actions are selected, as this might result in doing actions on unnecessary files/cloud objects. For more information on Recursion Depth, you can refer to the official Rclone documentation:
  • Copy Objects
  • Copy And Rename
  • Move Objects
  • Move And Rename
  • Delete Object
  • Delete Empty Directory
  • Synchronize


The default value is 1.

Introduced in 1.0.0
Error On No File TransferCheckbox

When enabled and when no files are transferred a failure exit code with value equal to 21 is raised.

Optional when one of the Two storage actions is selected.


The default setting is unchecked.

Introduced in 1.0.0
Dry-RunCheckboxWhen enabled, performs a trial run with no permanent changes of the selected Action.

Note: It is recommended to execute a Dry-Run as a test prior to the final task configuration when the following Actions are used:
  • Copy Objects
  • Copy And Rename
  • Move Objects
  • Move And Rename
  • Delete Object
  • Delete Empty Directory
  • Synchronize


The default value is unchecked.

Introduced in 1.0.0
Additional OptionsTextSpace-separated Rclone options applied to the selected Action.Introduced in 1.0.0
Log FormatChoiceOption for STDERR logging format.

Available options:
  • Text (default)
  • JSON
Introduced in 1.0.0

Output Fields


NameTypeDescriptionVersion Information
ProgressLarge TextLive progress information of the transferred files.Introduced in 1.0.0

Exit Codes

The exit codes for this Universal Extension are described below.

Exit CodeStatusStatus Description

0SuccessTask executed successfully.
1FailedExecution Failed: "<Error Description>"
20FailedData Validation Error: "<Error Description>"
21FailedExecution Failed: No Files were transferred.
22FailedExecution Failed: Task executed successfully, but updating Credentials failed.

STDOUT and STDERR


STDOUT and STDERR provide additional information to the User. The populated content can be changed in future versions of this extension without notice. Backward compatibility is not guaranteed.


Observability

Metrics are generated and exported by this integration, contributing to the UAC data observability, delivered since Universal Automation Center 7.5.0.0.

Metric: ue.cdt.rclone.speed

Name

Instrument Type

Unit (UCUM)

Attributes

Instrumented By

Description

ue.cdt.rclone.speed

Histogram

By/s

As defined on Metric Attributes.

Universal Controller (Through Universal Event Templates)

The average speed at which rclone transfers files.

When the transfer time is too small, the speed becomes too big (Bytes/Sec) and is displayed as 0 by RClone. In such cases, the transfer speed statistic is ignored and no relevant metric is sent.

Metric: ue.cdt.rclone.duration

Name

Instrument Type

Unit (UCUM)

Attributes

Instrumented By

Description

ue.cdt.rclone.duration

Histogram

s

As defined on Metric Attributes.

Universal Controller (Through Universal Event Templates)

The time took to complete the entire file transfer process, from initiation to completion.

Metric: ue.cdt.rclone.transfers

Name

Instrument Type

Unit (UCUM)

Attributes

Instrumented By

Description

ue.cdt.rclone.transfers

Histogram

{transfers}

As defined on Metric Attributes.

Universal Controller (Through Universal Event Templates)

The total number of file transfers.

Metric Attributes

The following attributes are applicable for ue.cdt.rclone.speed, ue.cdt.rclone.duration and ue.cdt.rclone.transfers.

Attribute Name

Enabled

Description

action

By Default

The selected action of the Universal Task.

source

By Default

The source storage of the files.

target

By Default

The source file path.

Other Observability Configuration Options

Administrators can update the Universal Event Template to control additional attributes that can be sent out for Universal Event-based Metrics. However, this configuration should be approached with caution. When the possible distinct values of those attributes are high, that might lead to high cardinality issues.

Administrators can activate them with caution as follows:

  1. Go to Administration → Universal Templates

  2. Select the Universal Template

  3. Click “Event Templates” Tab

  4. Select the Name of the Event Template you want to update, which represents the metric you are interested in.

  5. Update the “Optional Metric Labels” List the required Metric Labels.

How To

Import Universal Template

To use the Universal Template, you first should perform the following steps.

  1. This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.

  2. To import the Universal Template into your Controller, follow the instructions from the official documentation: Import An Integration.

  3. When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list.

Modifications of this integration, applied by users or customers, before or after import, might affect the supportability of this integration. For more information refer to Integration Modifications.

Configure Universal Task

To configure a new Universal Task, there are three steps required:

  • Create required Resolvable Credentials. Required as Input Fields on the Universal Task Configuration.
  • Create a new Script of type Data, for the Configuration File and populate it according to the following section of Setup Rclone Configuration File as UAC Script.
  • Create a new task, and enter the task-specific details that were created in the Universal Template.

Setup Rclone Configuration File as UAC Script

This integration depends on Rclone binary which needs a configuration file to locate the required information for the involved storage system(s). This file is passed from the Universal Controller to the Universal Agent as UAC Script through the Configuration File Input Field. The Universal Controller Script should be of type Data, and option Resolve UAC Variables should be enabled.

Integrated Storage Systems & Sample Configurations

Rclone is integrated with multiple storage systems. This integration has been tested against the following Storage Systems:

  • Amazon S3
  • Google Cloud Storage
  • Microsoft OneDrive Business
  • Local file system (Linux, Windows)
  • HTTP(S) URL


This integration should work properly against other Storage Systems as well, as long as the integration interface (see chapter Configure Universal Task) and functionalities listed in this document fulfill the needs of the Storage System.

For functionalities required for specific Storage Systems, users and customers are encouraged to open a Feature Request in our Customer Support Portal.


The indicative Rclone configuration examples below contain the basic connection parameters.

SystemRclone Configuration Documentation LinkExample
AWS S3AWS S3 Rclone Configuration Guide
Example with Credentials as UAC Functions
[aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_storage_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_storage_credentials}')}
region = us-east-2

[aws_s3_source]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_source_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_source_credentials}')}
region = us-east-2
acl = bucket-owner-full-control
role_arn = arn:aws:iam::<account_id>:role/<role policy>

[aws_s3_target]
type = s3
provider = AWS
env_auth = false
access_key_id = ${_credentialUser('${ops_ue_cloud_target_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_target_credentials}')}
region = us-east-2
Google Cloud Storage (Google Drive)Google Drive Rclone Configuration Guide
Example with Credentials as UAC Functions
[google_cloud_storage]
type = google cloud storage
service_account_file = ${_credentialPwd('${ops_ue_cloud_storage_credentials}')}
object_acl = bucketOwnerFullControl
project_number = johnprojectno
location = europe-west3

[google_cloud_storage_source]
type = google cloud storage
service_account_file = ${_credentialPwd('${ops_ue_cloud_source_credentials}')}
object_acl = bucketOwnerFullControl
project_number = johnprojectno
location = europe-west3

[google_cloud_storage_target]
type = google cloud storage
service_account_file = ${_credentialPwd('${ops_ue_cloud_target_credentials}')}
object_acl = bucketOwnerFullControl
project_number = johnprojectno
location = europe-west3
Microsoft OneDrive/Sharepoint 

Microsoft OneDrive/Sharepoint Configuration Guide

For One Drive / Sharepoint data transfers, refer also to Refresh Storage Credentials section.

Example (OneDrive) with Credentials as UAC Functions
[one_drive]
type = onedrive
token = ${_credentialToken('${ops_ue_cloud_storage_credentials}')}
client_id = ${_credentialUser('${ops_ue_cloud_storage_credentials}')}
client_secret = ${_credentialPassword('${ops_ue_cloud_storage_credentials}')}
drive_id = b!reXzf-NsJk-THISISASAMPLEDRIVEIDdfA_bM-DrjNN_hR5pD0u13Fvhi
drive_type = business

[one_drive_source]
type = onedrive
token = ${_credentialToken('${ops_ue_cloud_source_credentials}')}
client_id = ${_credentialUser('${ops_ue_cloud_source_credentials}')}
client_secret = ${_credentialPassword('${ops_ue_cloud_credentials}')}
drive_id = b!reXzf-NsJk-THISISASAMPLEDRIVEIDdfA_bM-DrjNN_hR5pD0u13Fvhi
drive_type = business

[one_drive_target]
type = onedrive
token = ${_credentialToken('${ops_ue_cloud_target_credentials}')}
client_id = ${_credentialUser('${ops_ue_cloud_target_credentials}')}
client_secret = ${_credentialPassword('${ops_ue_cloud_target_credentials}')}
drive_id = b!reXzf-NsJk-THISISASAMPLEDRIVEIDdfA_bM-DrjNN_hR5pD0u13Fvhi
drive_type = business
Local Filesystem 

Local Filesystem Configuration Guide

Example
[linux_source]
type = local
 
[linux_target]
type = local
 
[windows_source]
type = local
 
[windows_target]
type = local

Using Credentials

Storage system configuration embodies the related credentials for authentication and authorization. It is advised that account credentials, tokens, or any other essential information inside Rclone Configuration File to be passed as Universal Controller Credentials, through Credential functions to be kept safe. 

Task authors should be aware of Credential Attributes length limitations. For example, the 'Runtime Password' attribute of a Universal Controller Credential, supports strings with up to 512 characters, so it should not be used to store information larger than this value. 

There are several ways that Credential Functions can be used. To make the Configuration File immune to changes of credential names used in the task definition it is recommended to follow the Configuration through Credential Field Variables approach. Another benefit of this approach is that it is compatible with all the Universal Controller versions supported by this integration. However, alternative approaches are Configuration through Credential Name and Configuration through Credential as Variable, which rely on changes performed on the Configuration File itself, rather on the task definition. The next chapters provide more information on them. 

For local filesystem storage, no credentials are required in the Configuration File. In this case, the credentials that are provided in the Agent Details of the Universal Task are used. If not, the selected action is executed with the permissions of the user that runs the ubroker daemon.

Configuration through Credential Field Variables

The below table lists the variable that can be used for the respective input Credential Field and an example of a function that can be used in the Configuration File definition.

Field NameField VariableCredential Function Example
Storage Credentialsops_ue_cloud_storage_credentials${_credentialUser('${ops_ue_cloud_storage_credentials}')}
Source Credentialsops_ue_cloud_source_credentials${_credentialUser('${ops_ue_cloud_source_credentials}')}
Target Credentialsops_ue_cloud_target_credentials${_credentialUser('${ops_ue_cloud_target_credentials}')}


Example of a Configuration File for AWS S3 Storage

[aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_storage_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_storage_credentials}')}
region = us-east-2

[aws_s3_source]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_source_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_source_credentials}')}
region = us-east-2
acl = bucket-owner-full-control
role_arn = arn:aws:iam::<account_id>:role/<role policy>

[aws_s3_target]
type = s3
provider = AWS
env_auth = false
access_key_id = ${_credentialUser('${ops_ue_cloud_target_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_target_credentials}')}
region = us-east-2


Configuration through Credential Name

In this case, the Credential Name is used directly inside the Configuration File

Example of a Configuration File for AWS S3 Storage, using a Credential Function with the Credential Name

[aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('ue-aws-s3')}
secret_access_key = ${_credentialPwd('ue-aws-s3')}
region = us-east-2

When using this approach always have in mind that:

  1. Any update on the Storage/Source/Target Credential attributes will have no affect on the Configuration File. However if different Credential Names are used on the Task definition, then the configuration file needs to be updated as well to refer to the new Credential names
  2. A single change on the Configuration File will affect all the Cloud Data Transfer Tasks, which are utilizing this Script. 


Configuration through Credential as Variable


Starting from Universal Controller 7.6 and Cloud Data Transfer 4.0.0, Credentials can be provided as UAC Variables:



The above approach requires no custom changes in the Configuration File contents.

[aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_storage_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_storage_credentials}')}
region = us-east-2


Refresh Storage Credentials

Refresh token for Microsoft One Drive

Remote storages of type Microsoft One Drive & Sharepoint use OAuth2 token ({"access_token": ..., "token_type": ... , "refresh_token": ... , "expiry": .... }) to authorize users or applications. In this case and only when the access token has expired, Rclone exchanges the existing refresh token for a new access token and then establishes a connection with the remote storage. This integration provides the capability to save the new value of the token on the selected Universal Controller Credential, so it can be used and configured as a Credential Function.

To enable this functionality, the task input fields related to this configuration are Update Credentials, Controller URL, Controller Credentials. The integration will perform no Update Credentials actions if the identifies Storage/Source/Target System is not of type "onedrive".  


An example of such a configuration can be found below:



Refresh authorization fields for other remotes

Some remote storages might have a functionality of refreshing an authorization field. This integration provides the capability to save the new value of the authorization field on a selected Universal Controller Credential, so it can be used and configured as a Credential Function.

The task input fields related to this configuration are Update Credentials, Controller URL, Controller Credentials, and Refresh Storage Credentials


An example of such a configuration can be found below:


Import Grafana Dashboard

Users can benefit from a ready-to-use sample dashboard that this downloadable integration offers when observability features are used. It is located under the /observability/grafana/ directory inside the downloadable zip file from Stonebranch Integration Hub. Administrators should refer to the official Grafana User Guide on how to import a Grafana Dashboard.

Dashboard’s Prometheus data source is configured as a variable and thus needs to be mapped to an existing Data Source configured on the target Grafana instance.

Integration Modifications

Modifications applied by users or customers, before or after import, might affect the supportability of this integration. The following modifications are discouraged to retain the support level as applied for this integration.

  • Python code modifications should not be done.

  • Template Modifications

    • General Section

      • "Name", "Extension", "Variable Prefix", and "Icon" should not be changed.

    • Universal Template Details Section

      • "Template Type", "Agent Type", "Send Extension Variables", and "Always Cancel on Force Finish" should not be changed.

    • Result Processing Defaults Section

      • Success and Failure Exit codes should not be changed.

      • Success and Failure Output processing should not be changed.

    • Fields Restriction Section
      The setup of the template does not impose any restrictions, However with respect to the "Exit Code Processing Fields" section.

      1. Success/Failure exit codes need to be respected.

      2. In principle, as STDERR and STDOUT outputs can change in follow-up releases of this integration, they should not be considered as a reliable source for determining the success or failure of a task.

Event Template configuration related to “Metric Label Attributes” & “Optional Metric Labels” is allowed. However, administrators should be cautious of high cardinality scenarios that might occur

Users and customers are encouraged to report defects, or feature requests at Stonebranch Support Desk.

Migration Guide

Upgrading from version 3.0 to 4.0

When upgrading major versions, the following actions should be performed to migrate properly task definitions due to the breaking changes:

  • Tasks which reference Cloud Data Transfer task instance variables (for example sibling tasks within a Workflow), should be updated to reference the Cloud Data Transfer variables as ${ops_ue_cloud_<field_name>}.
  • Configuration File should be updated when credential fields are referenced:
Cloud Data Transfer 3.0.0 (and earlier)Cloud Data Transfer 4.0.0 (and later)
Example with variable prefix ue_cloud_dt
[aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_dt_storage_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_dt_storage_credentials}')}
region = us-east-2

[aws_s3_source]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_dt_source_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_dt_source_credentials}')}
region = us-east-2
acl = bucket-owner-full-control
role_arn = arn:aws:iam::<account_id>:role/<role policy>

[aws_s3_target]
type = s3
provider = AWS
env_auth = false
access_key_id = ${_credentialUser('${ops_ue_cloud_dt_target_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_dt_target_credentials}')}
region = us-east-2
Example with variable prefix ue_cloud
[aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_storage_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_storage_credentials}')}
region = us-east-2

[aws_s3_source]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_source_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_source_credentials}')}
region = us-east-2
acl = bucket-owner-full-control
role_arn = arn:aws:iam::<account_id>:role/<role policy>

[aws_s3_target]
type = s3
provider = AWS
env_auth = false
access_key_id = ${_credentialUser('${ops_ue_cloud_target_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_target_credentials}')}
region = us-east-2



Document References

This document references the following documents.

Document LinkDescription
Universal TemplatesUser documentation for creating Universal Templates in the Universal Controller user interface.
Universal TasksUser documentation for creating Universal Tasks in the Universal Controller user interface.
RcloneRclone official documentation.
Rclone InstallInstallation instructions.
Rclone DownloadsDownload links for Linux, Windows, and MacOS.
Rclone Storage Systems configurationConfiguration file details.
Rclone Exit CodesRclone exit codes list.

Known Issues

On version 4.0.1 only the agent user (the user running the Universal Agent) could run this integration on Linus OS.

On version 4.0.2 this limitation can be removed manually. To allow users other than the agent user to run a Cloud Data Transfer Task, follow these steps:

  1. Identify the Linux Agents on which the Cloud Data Transfer Task needs to run.

  2. Import the new version of ue-cloud-dt (4.0.2) on the desired Universal Controller.
  3. Initiate a Test Cloud Data Transfer Task (such as listing files on a local system) on the agents identified in step (1), using the agent user (thus leaving the Agent Details → Credentials field empty). This test ensures that the integration installs correctly on each agent, enabling future tasks to run smoothly under various user accounts.

Changelog

ue-cloud-dt-4.0.2 (2024-10-31)

Fixes

  • Fix permission issue on Linux OS when task is executed with a user different than the agent user (#42860)

ue-cloud-dt-4.0.1 (2024-08-08)

Fixes

  • Avoid setting executable flag for rclone binary on Windows (#41574)

ue-cloud-dt-4.0.0 (2024-06-21)

Enhancements

  • Added: New input fields Storage Credentials Variable, Source Credentials Variable, Target Credentials Variable, are supported starting from Universal Controller 7.6.0.0 (#41027).
  • Added: Support for automatic refresh of Token Credential field on Universal Controller, for Storage Systems of type One Drive (#40913).

Fixes

  • Fixed: Metrics not produced when Task's Log Level is INFO (#36684).
  • Fixed: Traceback was visible in STDERR for handled exceptions (#35731).
  • Fixed: Set exit code 22 when error is occurs during Update Credentials actions (#30107).

Deprecations and Breaking Changes

  • Breaking Change: Bundle third party dependencies within the integration package. This change forces the minimum supported Universal Controller and Universal Agent versions to be 7.4.0.0 (#36182).

  • Breaking Change: Variable prefix of the Universal Template is changed from ue_cloud_dt to ue_cloud. This change affects the Configuration File contents and referencing task instance variables from other tasks. (#36365).

ue-cloud-dt-3.0.0 (2023-10-07)

Enhancements

  • Changed: The progress bar, available on Universal Controllers of version 7.3 and above, now syncs with RClone's transfer completion (#34332).

  • Changed: Improved clarity of RClone's log messages with respect to log level (#34332).
  • Added: Introduced Observability metrics enabling enhanced visibility and understanding of behavior for Task Instances. This feature is enabled on Universal Controllers of version 7.5 and above (#34330).

  • Added: Filter options are available for all list actions (#34455).

Deprecations and Breaking Changes

  • Breaking Change: Updated the status descriptions to be more informative (#34335).

    Tasks or workflows evaluating the "Status Description" of the task Instance, either programmatically or within UAC, might be affected by it. In that case, they need to conform to the new "Status Description" Text

  • Breaking Change: Extension Invocation Fields > Some fields are renamed for better readability (#34335).

  • Breaking Change: Extension Invocation Fields > Credential fields are displayed as an object for better readability (#34335).

Fixes

  • Fixed: RClone's command will no longer include duplicate flags (#34265).

Other

  • Extra: The extension is supplemented with a Grafana dashboard that, by utilizing the produced observability metrics, calculates and displays statistical data (#34338).

ue-cloud-dt-2.0.1 (2023-07-28)

Fixes

  • Fixed: Retrieving STDOUT/STDERR might block a Task Instance in Running status (#33732).

ue-cloud-dt-2.0.0 (2022-10-26)

Deprecations and Breaking Changes

  • Breaking Change: Universal Template Name rename.
    This version of Universal Extension will have no effect on existing tasks created with Universal Template Cloud Data Transfer and extension archive ue-cloud-ft-1.0.0. Future updates on this integration will take place on top of ue-cloud-dt-2.0.0.

ue-cloud-ft-1.0.0 (2022-09-02)

Enhancements

  • Added: Initial version (#28787)