Disclaimer

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at Stonebranch Integration Hub - Terms of Use.

...

Refer to changelogChangelog for version history information.

Overview

This integration provides the capability to perform data transfers between cloud-based storage services and local or distributed file systems. It also provides data storage management capabilities like listing, creating, or deleting data storage objects. 

Key Features

This integration is equipped with the following key features.

...

...

1. This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.

2. To import the Universal Template into your Controller, follow the instructions from the official documentation: Import An Integration.

3. When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list.

Modifications of this integration, applied by users or customers, before or after import, might affect the supportability of this integration. For more information refer to Integration Modifications.

Configure Universal Task

To configure a new Universal Task, there are three steps required:

• Create required Resolvable Credentials. Required as Input Fields on the Universal Task Configuration.
• Create a new Script of type Data, for the Configuration File and populate it according to the following section of Setup Rclone Configuration File as UAC Script.
• Create a new task, and enter the task-specific details that were created in the Universal Template.

...

...

[aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_storage_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_storage_credentials}')}
region = us-east-2

[aws_s3_source]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_source_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_source_credentials}')}
region = us-east-2
acl = bucket-owner-full-control
role_arn = arn:aws:iam::<account_id>:role/<role policy>

[aws_s3_target]
type = s3
provider = AWS
env_auth = false
access_key_id = ${_credentialUser('${ops_ue_cloud_target_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_target_credentials}')}
region = us-east-2

[google_cloud_storage]
type = google cloud storage
service_account_file = ${_credentialPwd('${ops_ue_cloud_storage_credentials}')}
object_acl = bucketOwnerFullControl
project_number = johnprojectno
location = europe-west3

[google_cloud_storage_source]
type = google cloud storage
service_account_file = ${_credentialPwd('${ops_ue_cloud_source_credentials}')}
object_acl = bucketOwnerFullControl
project_number = johnprojectno
location = europe-west3

[google_cloud_storage_target]
type = google cloud storage
service_account_file = ${_credentialPwd('${ops_ue_cloud_target_credentials}')}
object_acl = bucketOwnerFullControl
project_number = johnprojectno
location = europe-west3

[one_drive]
type = onedrive
client_idtoken = ${_credentialUsercredentialToken('${ops_ue_cloud_storage_credentials}')}
client_secretid = ${_credentialPasswordcredentialUser('${ops_ue_cloud_storage_credentials}')}
driveclient_idsecret = ${_credentialUsercredentialPwd('${ops_ue_cloud_storage_credentials}')}
drive_typeid = business
token = ${_credentialToken('${ops_ue_cloud_storage_credentials}')}

b!reXzf-NsJk-THISISASAMPLEDRIVEIDdfA_bM-DrjNN_hR5pD0u13Fvhi
drive_type = business

[one_drive_source]
type = onedrive
token = ${_credentialToken('${ops_ue_cloud_source_credentials}')}
client_id = ${_credentialUser('${ops_ue_cloud_source_credentials}')}
client_secret = ${_credentialPasswordcredentialPwd('${ops_ue_cloud_source_credentials}')}
drive_id = ${_credentialUser('${ops_ue_cloud_source_credentials}')}
drive_b!reXzf-NsJk-THISISASAMPLEDRIVEIDdfA_bM-DrjNN_hR5pD0u13Fvhi
drive_type = business

[one_drive_target]
type = businessonedrive
token = ${_credentialToken('${ops_ue_cloud_storagetarget_credentials}')}
 
[one_drive_target]
type = onedrive
client_id = ${_credentialUser('${ops_ue_cloud_target_credentials}')}
client_secret = ${_credentialPasswordcredentialPwd('${ops_ue_cloud_target_credentials}')}
drive_typeid = business
token = ${_credentialToken('${ops_ue_cloud_target_storage_credentials}')}b!reXzf-NsJk-THISISASAMPLEDRIVEIDdfA_bM-DrjNN_hR5pD0u13Fvhi
drive_type = business

[linux_source]
type = local
 
[linux_target]
type = local
 
[windows_source]
type = local
 
[windows_target]
type = local

...

Storage system configuration embodies the related credentials for authentication and authorization. It is advised that account credentials, tokens, or any other essential information inside Rclone Configuration File to be passed as Universal Controller Credentials, through Credential functions to be kept safe. 

There are several ways that Credential Functions can be used. To make the Configuration File immune to changes of credential names used in the task definition it is recommended to follow the Configuration through Credential Field Variables approach. Another benefit of this approach is that it is compatible with all the Universal Controller versions supported by this integration. However, alternative approaches are Configuration through Credential Name and Configuration through Credential as Variable, which rely on changes performed on the Configuration File itself, rather on the task definition. The next chapters provide more information on them. 

...

The task input fields related to this configuration are Update Credentials, Controller URL, Controller Credentials, and Refresh Storage Credentials. 

An example of such a configuration can be found below:

...

When upgrading major versions, the following actions should be performed to migrate properly task definitions due to the breaking changes:

• Tasks which reference Cloud Data Transfer task instance variables (for example sibling tasks within a Workflow), should be updated to reference the Cloud Data Transfer variables as ${ops_ue_cloud_<field_name>}.
• Configuration File should be updated when credential fields are referenced:

[aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_dt_storage_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_dt_storage_credentials}')}
region = us-east-2

[aws_s3_source]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_dt_source_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_dt_source_credentials}')}
region = us-east-2
acl = bucket-owner-full-control
role_arn = arn:aws:iam::<account_id>:role/<role policy>

[aws_s3_target]
type = s3
provider = AWS
env_auth = false
access_key_id = ${_credentialUser('${ops_ue_cloud_dt_target_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_dt_target_credentials}')}
region = us-east-2

[aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_storage_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_storage_credentials}')}
region = us-east-2

[aws_s3_source]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_source_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_source_credentials}')}
region = us-east-2
acl = bucket-owner-full-control
role_arn = arn:aws:iam::<account_id>:role/<role policy>

[aws_s3_target]
type = s3
provider = AWS
env_auth = false
access_key_id = ${_credentialUser('${ops_ue_cloud_target_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_target_credentials}')}
region = us-east-2

Document References

This document references the following documents.

Changelog

...

Known Issues

On version 4.0.1 only the agent user (the user running the Universal Agent) could run this integration on Linux OS.

On version 4.0.2 this limitation can be removed manually. To allow users other than the agent user to run a Cloud Data Transfer Task, follow these steps:

1. Identify the Linux Agents on which the Cloud Data Transfer Task needs to run.

2. Import the new version of ue-cloud-dt (4.0.2) on the desired Universal Controller.

3. Initiate a Test Cloud Data Transfer Task (such as listing files on a local system) on the agents identified in step (1), using the agent user (thus leaving the Agent Details → Credentials field empty). This test ensures that the integration installs correctly on each agent, enabling future tasks to run smoothly under various user accounts.

Changelog

ue-cloud-dt-4.0.2 (2024-10-31)

Fixes

• Fix permission issue on Linux OS when task is executed with a user different than the agent user (#42860)

ue-cloud-dt-4.0.1 (2024-08-08)

Fixes

• Avoid setting executable flag for rclone binary on Windows (#41574)

...

• Added: New input fields Storage Credentials Variable, Source Credentials Variable, Target Credentials Variable, are supported starting from Universal Controller 7.6.0.0 (#41027).
• Added: Support for automatic refresh of Token Credential field on Universal Controller, for Storage Systems of type One Drive (#40913).

Fixes

• Fixed: Metrics not produced when Task's Log Level is INFO (#36684).
• Fixed: Traceback was visible in STDERR for handled exceptions (#35731).
• Fixed: Set exit code 22 when error is occurs during Update Credentials actions (#30107).

...

• Breaking Change: Updated the status descriptions to be more informative (#34335).

  Warning
  Tasks or workflows evaluating the "Status Description" of the task Instance, either programmatically or within UAC, might be affected by it. In that case, they need to conform to the new "Status Description" Text

  
  

• Breaking Change: Extension Invocation Fields > Some fields are renamed for better readability (#34335).

• Breaking Change: Extension Invocation Fields > Credential fields are displayed as an object for better readability (#34335).

Fixes

• Fixed: RClone's command will no longer include duplicate flags (#34265).

...

ue-cloud-dt-2.0.1 (2023-07-28)

Fixes

• Fixed: Retrieving STDOUT/STDERR might block a Task Instance in Running status (#33732).

...