Disclaimer

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at Stonebranch Integration Hub - Terms of Use.

Version Information

...

...

Refer to Changelog for version history information.

Overview

This integration provides the capability to perform data transfers between cloud-based storage services and local or distributed file systems. It also provides data storage management capabilities like listing, creating, or deleting data storage objects. 

Key Features

This integration is equipped with the following key features.

...

Requirements

This integration requires a Universal Agent, a Python runtime to execute the Universal Task, and an Rclone executable.

Supported Actions

This integration

...

Supported Actions

...

This integration supports multiple actions which can be grouped into the following categories: 

...

supports multiple actions which can be grouped into the following categories: 

Action Type: Single Storage Action

Single storage actions involve listing and managing existing data objects in a local or remote storage. 

Configuration Examples

Action Output

{
	"exit_code": 0,
	"status_description": "Task executed successfully",
	"invocation": {
		"extension": "ue-cloud-dt",
		"version": "

4.0.0",
		"fields": {
			"action": "List Directories",
             ...
		}
	},
	"changed": false,
	"result": {
		"cmd": "rclone lsd ue_aws_s3:john-drive/

--config=C:\\DATA\\tmp\\54cd05b2-1528-46d2-8f9c-

 8bc09dfdbf4.conf 

--log-level=DEBUG --max-depth=4 --use-json-log --exclude ue_execution/compiled",
		"rc": 0
	}
}

{
    "exit_code": 1,
    "status_description": "Execution Failed",
    "invocation": {
        "extension": "ue-cloud-dt",
        "version": "

4.0.0",
        "fields": {
             "action": "List Directories",
              ...
         },
    "changed": false,
    "result": {
        "cmd": "rclone ls ue_aws_s3:john-drive/
--config=/home//ee1152e6-2afe-4383-b611-dfe4eb3c7fde.con 
--log-level=DEBUG --max-depth=1 --use-json-log",
        "rc": 1
    }
}

-1 2023-08-30 13:24:22 -1 analyses
-1 2023-08-11 15:31:44 -1 logs
-1 2023-07-10 13:59:55 -1 macros
-1 2023-07-10 13:59:55 -1 models
-1 2023-07-10 13:59:55 -1 seeds
-1 2023-07-10 13:59:55 -1 snapshots
-1 2023-07-10 13:59:55 -1 tests
-1 2023-07-10 13:59:55 -1 ue_execution

Action Type: Two Storage Action

Two storage actions involve data transfers between local or cloud storages.  

Configuration Examples

Action Output

{
     "exit_code": 0,
     "status_description": "Task executed successfully",
     "invocation": {
          "extension": "ue-cloud-dt",
          "version": "

4.0.0",
          "fields": {
              "action": "Copy Objects",
              ...
          }
      },
      "changed": true,
      "result": {
          "cmd": "rclone copy local_ua:C:\\Users\\test\\Desktop\\dbt one_drive_target:john-drive 

--config=C:\\DATA\\tmp\\74f533e1-07c5-46f6-a48e-698aa55c7dd7.conf 

--log-level=DEBUG --max-depth=1 --progress --no-traverse --ignore-times --ignore-size 

--use-json-log --s3-no-check-bucket"

          "rc": 0
      }
}  

{
    "exit_code": 1,
    "status_description": "Execution Failed",
     "invocation": {
          "extension": "ue-cloud-dt",
          "version": "

4.0.0",
          "fields": {
              "action": "Copy Objects",
              ...
          }
      },
      "changed": true,
      "result": {
          "cmd": "rclone copy local_ua:C:\\Users\\test\\Desktop\\dbt one_drive_target:john-drive

 --config=C:\\DATA\\tmp\\74f533e1-07c5-46f6-a48e-698aa55c7dd7.conf 

--log-level=DEBUG --max-depth=1 --progress --no-traverse --ignore-times 

--ignore-size --use-json-log --s3-no-check-bucket",
          "rc": 1
      }
}  

Transferred:                          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:                       1.3sTransferred: 29 B / 300.002 MiB, 0%, 0 B/s, ETA -
Transferred:                          0 / 7, 0%
Elapsed time:                          1.8s
Transferring:
*                                   .gitignore:100% /29, 0/s, -
*                                   100MB.bin: 0% /100Mi, 0/s, -
*                                   100MB2.bin: 0% /100Mi, 0/s, -
*                                   100MB3.bin: 0% /100Mi, 0/s, -Transferred: 3.489 MiB 
/ 300.002 MiB, 1%, 0 B/s, ETA -
Transferred:                        1 / 7, 14%
Elapsed time:                      2.3s
Transferring:
*                                  100MB.bin: 1% /100Mi, 0/s, -
*                                  100MB2.bin: 0% /100Mi, 0/s, -
*                                  100MB3.bin: 0% /100Mi, 0/s, -
*                                  README.md:100% /571, 0/s, -Transferred: 18.834 MiB 
/ 300.002 MiB, 6%, 4.572 MiB/s,
ETA 1m1s
Transferred:                    2 / 7, 29%
Elapsed time:                   2.8s

Input Fields 

...

Output Fields

Exit Codes

...

STDOUT and STDERR

Observability

Metrics are generated and exported by this integration, contributing to the UAC data observability, delivered since Universal Automation Center 7.5.0.0.

Metric: ue.cdt.rclone.speed

...

Metric: ue.cdt.rclone.duration

Metric: ue.cdt.rclone.transfers

Metric Attributes

The following attributes are applicable for ue.cdt.rclone.speed, ue.cdt.rclone.duration and ue.cdt.rclone.transfers.

Other Observability Configuration Options

Administrators can update the Universal Event Template to control additional attributes thatcan be sent out for Universal Event-based Metrics. However, this configuration should be approached with caution. When the possible distinct values of those attributes are high, that might lead to high cardinality issues.

...

1. Go to Administration → Universal Templates

2. Select the Universal Template

3. Click “Event Templates” Tab

4. Select the Name of the Event Template you want to update, which represents the metric you are interested in.

5. Update the “Optional Metric Labels” List the required Metric Labels.

How To

Import Universal Template

To use the Universal Template, you first should perform the following steps.

...

Modifications of this integration, applied by users or customers, before or after import, might affect the supportability of this integration. For more information refer to Integration Modifications.

Configure Universal Task

To configure a new Universal Task, there are three steps required:

• Create required Resolvable Credentials. Required as Input Fields on the Universal Task Configuration.
• Create a new Script of type Data, for the Configuration File and populate it according to the following section of Setup Rclone Configuration File as UAC Script.
• Create a new task, and enter the task-specific details that were created in the Universal Template.

Setup Rclone Configuration File as UAC Script

This integration depends on Rclone binary which needs a configuration file to locate the required information for the involved storage system(s). This file is passed from the Universal Controller to the Universal Agent as UAC Script through the Configuration File Input Field. The Universal Controller Script should be of type Data, and option Resolve UAC Variables should be enabled.

Integrated Storage Systems & Sample Configurations

Rclone is integrated with multiple storage systems. This integration has been tested against the following Storage Systems:

...

The indicative Rclone configuration examples below contain the basic connection parameters.

[aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_

storage_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud

_storage_credentials}')}
region = us-east-2

[aws_s3_source]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud

_source_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud

_source_credentials}')}
region = us-east-2
acl = bucket-owner-full-control
role_arn = arn:aws:iam::<account_id>:role/<role policy>

[aws_s3_target]
type = s3
provider = AWS
env_auth = false
access_key_id = ${_credentialUser('${ops_ue_cloud

_target_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud

_target_credentials}')}
region = us-east-2

[google_cloud_storage]
type = google cloud storage
service_account_file = ${_credentialPwd('${ops_ue_cloud_

storage_credentials}')}
object_acl = bucketOwnerFullControl
project_number = johnprojectno
location = europe-west3

[google_cloud_storage_source]
type = google cloud storage
service_account_file = ${_credentialPwd('${ops_ue_cloud

_source_credentials}')}
object_acl = bucketOwnerFullControl
project_number = johnprojectno
location = europe-west3

[google_cloud_storage_target]
type = google cloud storage
service_account_file = ${_credentialPwd('${ops_ue_cloud

_target_credentials}')}
object_acl = bucketOwnerFullControl
project_number = johnprojectno
location = europe-west3

[one_drive]
type = onedrive
client_id = ${_credentialUser('${ops_ue_cloud_

storage_credentials}')}
client_secret = ${_credentialPassword('${ops_ue_cloud

_storage_credentials}')}
drive_id = ${_credentialUser('${ops_ue_cloud

_storage_credentials}')}
drive_type = business
token = ${_credentialToken('${ops_ue_cloud_

storage_credentials}')}

 
[one_drive_source]
type = onedrive
token = ${_credentialToken('${ops_ue_cloud

_source_credentials}')}
client_id = ${_credentialUser('${ops_ue_cloud

_source_credentials}')}
client_secret = ${_credentialPassword('${ops_ue_cloud

_source_credentials}')}
drive_id = ${_credentialUser('${ops_ue_cloud

_source_credentials}')}
drive_type = business
token = ${_credentialToken('${ops_ue_cloud

_storage_credentials}')}
 
[one_drive_target]
type = onedrive
client_id = ${_credentialUser('${ops_ue_cloud

_target_credentials}')}
client_secret = ${_credentialPassword('${ops_ue_cloud

_target_credentials}')}
drive_type = business
token = ${_credentialToken('${ops_ue_cloud_

target_storage_credentials}')}

[linux_source]
type = local
 
[linux_target]
type = local
 
[windows_source]
type = local
 
[windows_target]
type = local

Using Credentials

Storage system configuration embodies the related credentials for authentication and authorization. It is advised that account credentials, tokens, or any other essential information inside Rclone Configuration File to be passed as Universal Controller Credentials, through Credential functions to be kept safe. 

...

There are several ways that Credential Functions can be used. To make the Configuration File immune to changes of credential names used in the task definition it is recommended to follow the Configuration through Credential Field Variables approach. Another benefit of this approach is that it is compatible with all the Universal Controller versions supported by this integration. However, alternative approaches are Configuration through Credential Name and Configuration through Task Variables that contain the Credential NameCredential as Variable, which rely on changes performed on the Configuration File itself, rather on the task definition. The next chapters provide more information on them. 

For local filesystem storage, no credentials are required in the Configuration File. In this case, the credentials that are provided in the Agent Details of the Universal Task are used. If not, the selected action is executed with the permissions of the user that runs the ubroker daemon.

Configuration through Credential Field Variables

...

Example of a Configuration File for AWS S3 Storage

[aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_dt_storage_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_dt_storage_credentials}')}
region = us-east-2

[aws_s3_source]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_dt_source_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_dt_source_credentials}')}
region = us-east-2
acl = bucket-owner-full-control
role_arn = arn:aws:iam::<account_id>:role/<role policy>

[aws_s3_target]
type = s3
provider = AWS
env_auth = false
access_key_id = ${_credentialUser('${ops_ue_cloud_dt_target_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_dt_target_credentials}')}
region = us-east-2

Configuration through Credential Name

In this case, the Credential Name is used directly inside the Configuration File

Image RemovedImage Added

Example of a Configuration File for AWS S3 Storage, using a Credential Function with the Credential Name

...

1. Any update on the Storage/Source/Target Credential attributes will have no affect on the Configuration File. However if different Credential Names are used on the Task definition, then the configuration file needs to be updated as well to refer to the new Credential names. 
2. A single change on the Configuration File will affect all the Cloud Data Transfer Tasks, which are utilizing this Script. 

Configuration through

...

Credential as Variable

In the Variable Tab of the Starting from Universal Controller 7.6 and Cloud Data Transfer configuration, create a new Variable. Its value should be the name of an existing Resolvable Credential: 

Image Removed

Example of a Configuration File for AWS S3 Storage, using a Credential Function with a variable that contains the Credential Name4.0.0, Credentials can be provided as UAC Variables:

Image Added

The above approach requires no custom changes in the Configuration File contents.

[aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${AWS_S3_Credentialsops_ue_cloud_storage_credentials}')}
secret_access_key = ${_credentialPwd('${AWS_S3_Credentialsops_ue_cloud_storage_credentials}')}
region = us-east-2

When using this approach always have in mind that:

1. Any update on the Storage/Source/Target Credentials will have no affect on the Configuration File. 
2. Both the Configuration File and the Cloud Data Transfer Task definition can remain intact, while the credential names are managed only by the Task Variables.

...

Refresh Storage Credentials

Refresh token for Microsoft One Drive

Some remote Remote storages like Microsoft of type Microsoft One Drive & Sharepoint, use Sharepoint use OAuth2 token ({"access_token": ..., "token_type": ... , "refresh_token": ... , "expiry": .... }) to authorize users or applications. In this case and only when the access token has expired, Rclone exchanges the existing refresh token for a new access token and then establishes a connection with the remote storage. This integration provides the capability to save the new value of the token on a the selected Universal Controller Credential, so it can be used and configured as a Credential Function.

The To enable this functionality, the task input fields related to this configuration are Update Credentials, Controller URL, Controller Credentials, and Refresh Storage Credentials. An example of such a configuration . The integration will perform no Update Credentials actions if the identifies Storage/Source/Target System is not of type "onedrive".  

An example of such a configuration can be found below:

Image RemovedImage Added

Import Grafana Dashboard

Users can benefit from a ready-to-use sample dashboard that this downloadable integration offers when observability features are used. It is located under the /observability/grafana/ directory inside the downloadable zip file from Stonebranch Integration Hub. Administrators should refer to the official Grafana User Guide on how to import a Grafana Dashboard.

Integration Modifications

...

Modifications applied by users or customers, before or after import, might affect the supportability of this integration. The following modifications are discouraged to retain the support level as applied for this integration.

...

Refresh authorization fields for other remotes

Some remote storages might have a functionality of refreshing an authorization field. This integration provides the capability to save the new value of the authorization field on a selected Universal Controller Credential, so it can be used and configured as a Credential Function.

The task input fields related to this configuration are Update Credentials, Controller URL, Controller Credentials, and Refresh Storage Credentials. 

An example of such a configuration can be found below:

Image Added

Import Grafana Dashboard

Users can benefit from a ready-to-use sample dashboard that this downloadable integration offers when observability features are used. It is located under the /observability/grafana/ directory inside the downloadable zip file from Stonebranch Integration Hub. Administrators should refer to the official Grafana User Guide on how to import a Grafana Dashboard.

Integration Modifications

Modifications applied by users or customers, before or after import, might affect the supportability of this integration. The following modifications are discouraged to retain the support level as applied for this integration.

• Python code modifications should not be done.

• Template Modifications

  • General Section

    • "Name", "Extension", "Variable Prefix", and "Icon" should not be changed.

  • Universal Template Details Section

    • "Template Type", "Agent Type", "Send Extension Variables", and "Always Cancel on Force Finish" should not be changed.

  • Result Processing Defaults Section

    • Success and Failure Exit codes should not be changed.

    • Success and Failure Output processing should not be changed.

  • Fields Restriction Section
    The setup of the template does not impose any restrictions, However with respect to the "Exit Code Processing Fields" section.

    1. Success/Failure exit codes need to be respected.

    2. In principle, as STDERR and STDOUT outputs can change in follow-up releases of this integration, they should not be considered as a reliable source for determining the success or failure of a task.

...

Users and customers are encouraged to report defects, or feature requests at Stonebranch Support Desk.

Document References

This document references the following documents.

...

Changelog

...

Migration Guide

Upgrading from version 3.0 to 4.0

When upgrading major versions, the following actions should be performed to migrate properly task definitions due to the breaking changes:

• Tasks which reference Cloud Data Transfer task instance variables (for example sibling tasks within a Workflow), should be updated to reference the Cloud Data Transfer variables as ${ops_ue_cloud_<field_name>}.
• Configuration File should be updated when credential fields are referenced:

[aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_dt_storage_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_dt_storage_credentials}')}
region = us-east-2

[aws_s3_source]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_dt_source_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_dt_source_credentials}')}
region = us-east-2
acl = bucket-owner-full-control
role_arn = arn:aws:iam::<account_id>:role/<role policy>

[aws_s3_target]
type = s3
provider = AWS
env_auth = false
access_key_id = ${_credentialUser('${ops_ue_cloud_dt_target_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_dt_target_credentials}')}
region = us-east-2

[aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_storage_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_storage_credentials}')}
region = us-east-2

[aws_s3_source]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_source_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_source_credentials}')}
region = us-east-2
acl = bucket-owner-full-control
role_arn = arn:aws:iam::<account_id>:role/<role policy>

[aws_s3_target]
type = s3
provider = AWS
env_auth = false
access_key_id = ${_credentialUser('${ops_ue_cloud_target_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_target_credentials}')}
region = us-east-2

Document References

This document references the following documents.

Changelog

ue-cloud-dt-4.0.0 (2024-06-21)

Enhancements

• Added: New input fields Storage Credentials Variable, Source Credentials Variable, Target Credentials Variable, are supported starting from Universal Controller 7.6.0.0 (#41027).
• Added: Support for automatic refresh of Token Credential field on Universal Controller, for Storage Systems of type One Drive (#40913).

Fixes

• Fixed: Metrics not produced when Task's Log Level is INFO (#36684).
• Fixed: Traceback was visible in STDERR for handled exceptions (#35731).
• Fixed: Set exit code 22 when error is occurs during Update Credentials actions (#30107).

Deprecations and Breaking Changes

• Breaking Change: Bundle third party dependencies within the integration package. This change forces the minimum supported Universal Controller and Universal Agent versions to be 7.4.0.0 (#36182).

• Breaking Change: Variable prefix of the Universal Template is changed from ue_cloud_dt to ue_cloud. This change affects the Configuration File contents and referencing task instance variables from other tasks. (#36365).

ue-cloud-dt-3.0.0 (2023-10-07)

Enhancements

• Changed: The progress bar, available on Universal Controllers of version 7.3 and above, now syncs with RClone's transfer completion (#34332).

• Changed: Improved clarity of RClone's log messages with respect to log level (#34332).

• Added: Introduced Observability metrics enabling enhanced visibility and understanding of behavior for Task Instances. This feature is enabled on Universal Controllers of version 7.5 and above (#34330).

• Added: Filter options are available for all list actions (#34455).

Deprecations and Breaking Changes

• Breaking Change: Updated the status descriptions to be more informative (#34335).

  Warning
  Tasks or workflows evaluating the "Status Description" of the task Instance, either programmatically or within UAC, might be affected by it. In that case, they need to conform to the new "Status Description" Text

  
  

• Breaking Change: Extension Invocation Fields > Some fields are renamed for better readability (#34335).

• Breaking Change: Extension Invocation Fields > Credential fields are displayed as an object for better readability (#34335).

Fixes

• Fixed: RClone's command will no longer include duplicate flags (#34265).

Other

• Extra: The extension is supplemented with a Grafana dashboard that, by utilizing the produced observability metrics, calculates and displays statistical data (#34338).

ue-cloud-dt-2.0.1 (2023-07-28)

Fixes

• Fixed: Retrieving STDOUT/STDERR might block a Task Instance in Running status (#33732).

ue-cloud-dt-2.0.0 (2022-10-26)

Deprecations and Breaking Changes

• Breaking Change: Universal Template Name rename.
  This version of Universal Extension will have no effect on existing tasks created with Universal Template Cloud Data Transfer and extension archive ue-cloud-ft-1.0.0. Future updates on this integration will take place on top of ue-cloud-dt-2.0.0.

ue-cloud-ft-1.0.0 (2022-09-02)

Enhancements

• Added: Initial version (#28787)

...