Panel

Table of Contents

maxLevel	2

Disclaimer

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at Stonebranch Integration Hub - Terms of Use.

...

Template Name	Extension Name	Version	Status
Cloud Data Transfer	ue-cloud-dt	4 (Current 4.0.12)	Fixes and new features are introduced.
Cloud Data Transfer	ue-cloud-dt	3	Hot Fixes Only (Until UAC 7.3 is End of Support)
Cloud Data Transfer	ue-cloud-dt	1 & 2	End of Support

...

Warning
Version 4.0.0 is a major release and introduces breaking changes that might affect some users depending on their setup. Administrators are strongly advised to refer to Changelog for more information on the changes introduced in this release. Refer also to Migration Guide to apply changes in existing task definitions.

Overview

This integration provides the capability to perform data transfers between cloud-based storage services and local or distributed file systems. It also provides data storage management capabilities like listing, creating, or deleting data storage objects.

Key Features

This integration is equipped with the following key features.

...

Name Type Description Version Information

Action

Choice

Action to be performed.

Available actions:

Copy Objects (default)
Copy And Rename
Copy From URL
Move Objects
Move And Rename
List Objects
List Directories
List Objects With Details
List Objects In JSON Format
Create Object
Delete Object
Delete Empty Directory
Synchronize

Introduced in 1.0.0

Configuration File

Script

This integration depends on Rclone. This field models the Rclone Configuration file, which contains all required parameters to connect to the Storage System, Source Storage and Target Storage. The filename should have the ".conf" suffix.

Example: rclone.conf

Introduced in 1.0.0

Storage System Dynamic Choice Field Remote or Local storage to execute the selected Action.
Execute the Dynamic Choice Field and retrieve all the available Storages from Configuration File.

Required when one of the Single storage actions is selected. Introduced in 1.0.0

Source Storage Dynamic Choice Field Remote or Local source storage, where transfer Action is initiated.
Execute the Dynamic Choice Field and retrieve automatically all the available Storages from Configuration File.

Required when one of the Two storage actions is selected. Introduced in 1.0.0

Target Storage Dynamic Choice Field Remote or Local target storage, where transferred data is stored.
Execute the Dynamic Choice Field and retrieve automatically all the available Storages from Configuration File.

Required when one of the Two storage actions is selected. Introduced in 1.0.0

Storage Credentials

Credential

Credentials needed to connect to the selected Storage System.
For Storage System of type local, the standard Agent's Credentials field should be provided.

From Universal Controller 7.6 onwards a variable that holds the Credential Name can be used when related checkbox is selected. In this case the content of this field should be ${variable_name}

Required when one of the Single storage actions is selected.

Introduced in 1.0.0

Source Credentials

Credential

Credentials needed to connect to the selected Source Storage.

From Universal Controller 7.6 onwards a variable that holds the Credential Name can be used when related checkbox is selected. In this case the content of this field should be ${variable_name}

Required when one of the Two storage actions is selected. For Storage System of type local, the standard Agent's Credentials field should be provided.

Introduced in 1.0.0

Target Credentials

Credential

Credentials needed to connect to the selected Target Storage.
From Universal Controller 7.6 onwards a variable that holds the Credential Name can be used when related checkbox is selected. In this case the content of this field should be ${variable_name}

Required when one of the Two storage actions is selected.

For Storage System of type local, the standard Agent's Credentials field should be provided.

Introduced in 1.0.0

Filepath Text File or directory path from where to retrieve data for the selected Action.

Required when one of the Single storage actions is selected. Introduced in 1.0.0

Source Filepath Text File or directory path from where to retrieve data for the selected Action.

Required when one of the Two storage actions is selected. Introduced in 1.0.0

Target Filepath Text File or directory path where transferred data will be stored for the selected Action.

Required when one of the Two storage actions is selected. Introduced in 1.0.0

Update Credentials

Checkbox

If the remote storage uses OAuth for its authorization, Rclone can be adjusted to refresh the respective token in its Configuration File during execution time. Enabling this field, will ensure that the runtime-updated token from Rclone, will get provisioned to the respective Credential Field on the Universal Controller.

This field is used in conjunction with fields Controller URL, Controller Credentials, and optionally the Refresh Storage Credentials. See the respective examples in Refresh Storage Credentials chapter.

The default setting is unchecked.

Introduced in 1.0.0

Controller URL Text Controller URL where Storage Credentials, Source Credentials, Target Credentials are stored.

Required when Update Credentials is checked. Introduced in 1.0.0

Controller Credentials

Credentials

Controller user's credentials, used for logging and updating Storage Credentials, Source Credentials, Target Credentials. The Credentials definition should be as follows:

User as "Runtime User".
User Password as "Runtime Password".

Note: The provided user should have sufficient Credential permissions.

Required when Update Credentials is checked.

Introduced in 1.0.0

Refresh Storage Credentials

Anchor

	Refresh Storage Credential
	Refresh Storage Credential

Array

Note: This Array is not required to be configured in case the Storage type is "One Drive".

This Array field is available when Update Credentials is checked. Is used as a mapping table between the Configuration File locator that has been updated by Rclone during execution, and its corresponding Universal Controller Credential field.

Populate this array with the following format:

Configuration File Details: "<Storage/Source/Target Storage>"."<Configuration File's field to be refreshed by Rclone>"
Storage Credentials Details: "<Storage/Source/Target Credentials name>"."<Storage/Source/Target Credentials field to be updated>"

The result of this action is that the Configuration File's locator value, is copied to the provided Storage Credential's field. Each line of this Array is evaluated, and related mapping is performed. See the Refresh authorization fields for other remotes.

Note: Refresh of Storage/Source/Target Credentials is performed via Controller's REST API. See Network Requirements .

Introduced in 1.0.0

Use Filter

Choice

The filter type that is applied on the Action.

Available options:

-- None – (default)
Include
Exclude

Optional for all, but Create Object Action.

Introduced in 1.0.0

Filter Text Filter as regular expression that is applied on the Action.

Required when Use Filter is checked. Introduced in 1.0.0

Overwrite Options

Choice

Options for overwriting files in the Target Storage.

Available options:

Overwrite
Do Not Overwrite (default)
Append Timestamp In Filename

Optional when one of the Two storage actions is selected.

Introduced in 1.0.0

Recursion Depth

Integer

Recursion depth that is applied. Default value 1 means that no recursion will be applied.

Note: Use this option with caution when one of the following Actions are selected, as this might result in doing actions on unnecessary files/cloud objects. For more information on Recursion Depth, you can refer to the official Rclone documentation:

Copy Objects
Copy And Rename
Move Objects
Move And Rename
Delete Object
Delete Empty Directory
Synchronize

The default value is 1.

Introduced in 1.0.0

Error On No File Transfer

Checkbox

When enabled and when no files are transferred a failure exit code with value equal to 21 is raised.

Optional when one of the Two storage actions is selected.

The default setting is unchecked.

Introduced in 1.0.0

Dry-Run

Checkbox

When enabled, performs a trial run with no permanent changes of the selected Action.

Note: It is recommended to execute a Dry-Run as a test prior to the final task configuration when the following Actions are used:

Copy Objects
Copy And Rename
Move Objects
Move And Rename
Delete Object
Delete Empty Directory
Synchronize

The default value is unchecked.

Introduced in 1.0.0

Additional Options Text Space-separated Rclone options applied to the selected Action. Introduced in 1.0.0

Log Format

Choice

Option for STDERR logging format.

Available options:

Text (default)
JSON

Introduced in 1.0.0

...

This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.
To import the Universal Template into your Controller, follow the instructions from the official documentation: Import An Integration.
When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list.

...

System

Rclone Configuration Documentation Link

Example

AWS S3

AWS S3 Rclone Configuration Guide

Code Block

language	text
title	Example with Credentials as UAC Functions
collapse	true

[aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_storage_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_storage_credentials}')}
region = us-east-2

[aws_s3_source]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_source_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_source_credentials}')}
region = us-east-2
acl = bucket-owner-full-control
role_arn = arn:aws:iam::<account_id>:role/<role policy>

[aws_s3_target]
type = s3
provider = AWS
env_auth = false
access_key_id = ${_credentialUser('${ops_ue_cloud_target_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_target_credentials}')}
region = us-east-2

Google Cloud Storage (Google Drive)

Google Drive Rclone Configuration Guide

Code Block

language	text
title	Example with Credentials as UAC Functions
collapse	true

[google_cloud_storage]
type = google cloud storage
service_account_file = ${_credentialPwd('${ops_ue_cloud_storage_credentials}')}
object_acl = bucketOwnerFullControl
project_number = johnprojectno
location = europe-west3

[google_cloud_storage_source]
type = google cloud storage
service_account_file = ${_credentialPwd('${ops_ue_cloud_source_credentials}')}
object_acl = bucketOwnerFullControl
project_number = johnprojectno
location = europe-west3

[google_cloud_storage_target]
type = google cloud storage
service_account_file = ${_credentialPwd('${ops_ue_cloud_target_credentials}')}
object_acl = bucketOwnerFullControl
project_number = johnprojectno
location = europe-west3

Microsoft OneDrive/Sharepoint

Microsoft OneDrive/Sharepoint Configuration Guide

For One Drive / Sharepoint data transfers, refer also to Refresh Storage Credentials section.

Code Block

language	text
title	Example (OneDrive) with Credentials as UAC Functions
collapse	true

[one_drive]
type = onedrive
client_idtoken = ${_credentialUsercredentialToken('${ops_ue_cloud_storage_credentials}')}
client_secretid = ${_credentialPasswordcredentialUser('${ops_ue_cloud_storage_credentials}')}
driveclient_idsecret = ${_credentialUsercredentialPwd('${ops_ue_cloud_storage_credentials}')}
drive_typeid = business
token = ${_credentialToken('${ops_ue_cloud_storage_credentials}')}

b!reXzf-NsJk-THISISASAMPLEDRIVEIDdfA_bM-DrjNN_hR5pD0u13Fvhi
drive_type = business

[one_drive_source]
type = onedrive
token = ${_credentialToken('${ops_ue_cloud_source_credentials}')}
client_id = ${_credentialUser('${ops_ue_cloud_source_credentials}')}
client_secret = ${_credentialPasswordcredentialPwd('${ops_ue_cloud_source_credentials}')}
drive_id = ${_credentialUser('${ops_ue_cloud_source_credentials}')}b!reXzf-NsJk-THISISASAMPLEDRIVEIDdfA_bM-DrjNN_hR5pD0u13Fvhi
drive_type = business
token 
[one_drive_target]
type = onedrive
token = ${_credentialToken('${ops_ue_cloud_storagetarget_credentials}')}
 
[one_drive_target]
type
= onedrive
client_id = ${_credentialUser('${ops_ue_cloud_target_credentials}')}
client_secret = ${_credentialPasswordcredentialPwd('${ops_ue_cloud_target_credentials}')}
drive_typeid = business
token = ${_credentialToken('${ops_ue_cloud_target_storage_credentials}')}b!reXzf-NsJk-THISISASAMPLEDRIVEIDdfA_bM-DrjNN_hR5pD0u13Fvhi
drive_type = business

Local Filesystem

Local Filesystem Configuration Guide

Code Block

language	text
title	Example
collapse	true

[linux_source]
type = local
 
[linux_target]
type = local
 
[windows_source]
type = local
 
[windows_target]
type = local

...

Storage system configuration embodies the related credentials for authentication and authorization. It is advised that account credentials, tokens, or any other essential information inside Rclone Configuration File to be passed as Universal Controller Credentials, through Credential functions to be kept safe.

Note
Task authors should be aware of Credential Attributes length limitations. For example, the 'Runtime Password' attribute of a Universal Controller Credential, supports strings with up to 512 characters, so it should not be used to store information larger than this value.

...

The task input fields related to this configuration are Update Credentials, Controller URL, Controller Credentials, and Refresh Storage Credentials.

An example of such a configuration can be found below:

...

Cloud Data Transfer 3.0.0 (and earlier)

Cloud Data Transfer 4.0.0 (and later)

Code Block

language	text
title	Example with variable prefix ue_cloud_dt
collapse	true

[aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_dt_storage_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_dt_storage_credentials}')}
region = us-east-2

[aws_s3_source]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_dt_source_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_dt_source_credentials}')}
region = us-east-2
acl = bucket-owner-full-control
role_arn = arn:aws:iam::<account_id>:role/<role policy>

[aws_s3_target]
type = s3
provider = AWS
env_auth = false
access_key_id = ${_credentialUser('${ops_ue_cloud_dt_target_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_dt_target_credentials}')}
region = us-east-2

Code Block

language	text
title	Example with variable prefix ue_cloud
collapse	true

[aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_storage_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_storage_credentials}')}
region = us-east-2

[aws_s3_source]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_source_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_source_credentials}')}
region = us-east-2
acl = bucket-owner-full-control
role_arn = arn:aws:iam::<account_id>:role/<role policy>

[aws_s3_target]
type = s3
provider = AWS
env_auth = false
access_key_id = ${_credentialUser('${ops_ue_cloud_target_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_target_credentials}')}
region = us-east-2

...

Document References

This document references the following documents.

Document Link	Description
Universal Templates	User documentation for creating Universal Templates in the Universal Controller user interface.
Universal Tasks	User documentation for creating Universal Tasks in the Universal Controller user interface.
Rclone	Rclone official documentation.
Rclone Install	Installation instructions.
Rclone Downloads	Download links for Linux, Windows, and MacOS.
Rclone Storage Systems configuration	Configuration file details.
Rclone Exit Codes	Rclone exit codes list.

Changelog

...

Known Issues

On version 4.0.1 only the agent user (the user running the Universal Agent) could run this integration on Linux OS.

On version 4.0.2 this limitation can be removed manually. To allow users other than the agent user to run a Cloud Data Transfer Task, follow these steps:

Identify the Linux Agents on which the Cloud Data Transfer Task needs to run.
Import the new version of ue-cloud-dt (4.0.2) on the desired Universal Controller.
Initiate a Test Cloud Data Transfer Task (such as listing files on a local system) on the agents identified in step (1), using the agent user (thus leaving the Agent Details → Credentials field empty). This test ensures that the integration installs correctly on each agent, enabling future tasks to run smoothly under various user accounts.

Changelog

Anchor

	Changelog
	Changelog

ue-cloud-dt-4.0.2 (2024-10-31)

Fixes

Fix permission issue on Linux OS when task is executed with a user different than the agent user (#42860)

ue-cloud-dt-4.0.1 (2024-08-08)

Fixes

Avoid setting executable flag for rclone binary on Windows (#41574)

...

Added: New input fields Storage Credentials Variable, Source Credentials Variable, Target Credentials Variable, are supported starting from Universal Controller 7.6.0.0 (#41027).
Added: Support for automatic refresh of Token Credential field on Universal Controller, for Storage Systems of type One Drive (#40913).

Fixes

Fixed: Metrics not produced when Task's Log Level is INFO (#36684).
Fixed: Traceback was visible in STDERR for handled exceptions (#35731).
Fixed: Set exit code 22 when error is occurs during Update Credentials actions (#30107).

...

Breaking Change: Updated the status descriptions to be more informative (#34335).

Warning
Tasks or workflows evaluating the "Status Description" of the task Instance, either programmatically or within UAC, might be affected by it. In that case, they need to conform to the new "Status Description" Text

Breaking Change: Extension Invocation Fields > Some fields are renamed for better readability (#34335).
Breaking Change: Extension Invocation Fields > Credential fields are displayed as an object for better readability (#34335).

Fixes

Fixed: RClone's command will no longer include duplicate flags (#34265).

...

ue-cloud-dt-2.0.1 (2023-07-28)

Fixes

Fixed: Retrieving STDOUT/STDERR might block a Task Instance in Running status (#33732).

...

Versions Compared

Old Version 44

New Version Current

Key

Disclaimer

Overview

Key Features

Document References

Changelog

Known Issues

Changelog

ue-cloud-dt-4.0.2 (2024-10-31)

Fixes

ue-cloud-dt-4.0.1 (2024-08-08)

Fixes

Fixes

Fixes

ue-cloud-dt-2.0.1 (2023-07-28)

Fixes

Page Comparison

Versions Compared

Old Version 44

New Version Current

Key

Disclaimer

Overview

Key Features

Document References

Changelog

Known Issues

Changelog

ue-cloud-dt-4.0.2 (2024-10-31)

Fixes

ue-cloud-dt-4.0.1 (2024-08-08)

Fixes

Fixes

Fixes

ue-cloud-dt-2.0.1 (2023-07-28)

Fixes