Disclaimer

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at https://www.stonebranch.com/integration-hub/Terms-and-Privacy/Terms-of-Use/

Overview

This Universal Extension is an interface to Rclone that provides the capability to manage and synchronize files from across different cloud storages, as well as local or distributed file systems. Rclone, is the open source command line program, that is utilized to accomplish all the actions supported in this Extension.

Version Information

Software Requirements

This integration requires a Universal Agent, a Python runtime to execute the Universal Task, and a Rclone executable.

Software Requirements for Universal Template and Universal Task

Requires Python 3.7 or higher. Tested with the Universal Agent bundled Python distribution.

Software Requirements for Universal Agent

Both Windows and Linux agents are supported.

• Universal Agent for Windows x64 Version 7.2.0.0 and later with Python options installed.
• Universal Agent for Linux Version 7.2.0.0 and later with Python options installed.

Software Requirements for Universal Controller

Universal Controller Version 7.2.0.0 and later.

Software Requirements for Rclone

Rclone needs to be installed on the same server where the Universal Agent is installed. Rclone binary should be stored on a location where Universal Agent has access and permissions to execute it.

Network and Connectivity Requirements

Universal Agent should be able to establish a connection with:

• The remote storages that are provided as input for the Universal Task.
• The Universal Controller host. This is required only in case Refresh Storage Credentials field is used.

Key Features

This Universal Extension is an interface for Rclone and as an interface, it supports the following key features:

• Actions
  • Copy, move, and synchronize data between two storages.
  • Copy a URL's content to cloud or local destination without saving it in temporary storage.
  • List data on a storage, including listing with details or in JSON format for machine parsing.
  • Create objects on a storage.
  • Delete objects from a storage.
• Features:
  • Fast transfers for objects stored in the same region.
  • Preserves always timestamps and verifies checksums.
  • Supports encryption, caching, compression, and chunking.
  • Dynamic token updates for OneDrive Business cloud storage, observing the OneDrive business refresh token flow.
  • Support for dry runs. Allows users to execute a Universal Task without making any permanent changes to the target storage.
  • Advanced filtering capability for files or objects to be listed or transferred.
  • Option to mark the Universal Task as Failed when no files have been transferred.
  • List of overwrite options for existing data.
  • Additional customized options.
  • Rich Observabilty capabilities.
• Output
  • Progress of the selected Action is visible, during Universal Task Instance execution.
  • Text or JSON formatted output.

Integrated Storage Systems

Rclone is integrated with multiple storage systems (an overview can be found here). This Universal Extension has been tested against the following Storage Systems.

• Amazon S3
• Google Cloud Storage
• Microsoft OneDrive Business
• Local file system (Linux, Windows)
• HTTP(S) URL

Import Universal Template

To use the Universal Template, you first should perform the following steps.

1. This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.

2. To import the Universal Template into your Controller, follow the instructions here.

3. When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list.

Modifications of this integration, applied by users or customers, before or after import, might affect the supportability of this integration. For more information refer to Integration Modifications.

Configure Universal Task

To configure a new Universal Task, there are two steps required:

• Create required Resolvable Credentials. Required as Input Fields on the Universal Task Configuration.
• Create a new Script of type Data, for the Configuration File and populate it according to the following section of Setup Rclone Configuration File. Required as Input Fields on the Universal Task Configuration.
• Create a new task, and enter the task-specific details that were created in the Universal Template.

Setup Rclone Configuration File

1. The configuration file follows the INI format and contains all required parameters and credentials to connect to a Storage System. All configuration options per Storage System can be found in the respective "Config Rclone documentation" here.

2. It is advised that account credentials, tokens, or any other essential information be passed as an encrypted Resolvable Credential field. To set a value with the respective Credential field, please make sure to use the appropriate Credential function inside the configuration file.

   'Runtime Password' Credentials field supports values with up to 512 characters.

   Hard-coded values can be provided in the script file, as well. However, it is not advised for security reasons.

3. It is recommended to configure a storage system section, one to one with the Credential fields in this Universal Extension (please refer to the Inter-Cloud Data Transfer#Input Fields section below).

4. For local file system Storages, no credentials are required in the configuration file. The selected action will be executed with the Credentials that are provided in the "Agent Details" of the Universal Task. If no Credentials are provided for the used Agent, then the same user that runs ubroker daemon will be used.

5. Script Type should be Data, and option Resolve UAC Variables should be enabled.

The indicative Configuration File below contains the basic connection parameters (flags):

[ue_aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_dt_storage_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_dt_storage_credentials}')}
region = us-east-2

[ue_aws_s3_source]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_dt_source_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_dt_source_credentials}')}
region = us-east-2
acl = bucket-owner-full-control
role_arn = arn:aws:iam::<account_id>:role/<role policy>

[ue_aws_s3_target]
type = s3
provider = AWS
env_auth = false
access_key_id = ${_credentialUser('${ops_ue_cloud_dt_target_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_dt_target_credentials}')}
region = us-east-2

[google_cloud_storage]
type = google cloud storage
service_account_file = ${_credentialPwd('${ops_ue_cloud_dt_storage_credentials}')}
object_acl = bucketOwnerFullControl
project_number = johnprojectno
location = europe-west3

[google_cloud_storage_source]
type = google cloud storage
service_account_file = ${_credentialPwd('${ops_ue_cloud_dt_source_credentials}')}
object_acl = bucketOwnerFullControl
project_number = johnprojectno
location = europe-west3

[google_cloud_storage_target]
type = google cloud storage
service_account_file = ${_credentialPwd('${ops_ue_cloud_dt_target_credentials}')}
object_acl = bucketOwnerFullControl
project_number = johnprojectno
location = europe-west3
 
[one_drive]
type = onedrive
token = ${_credentialToken('${ops_ue_cloud_dt_storage_credentials}')}
drive_id = ${_credentialUser('${ops_ue_cloud_dt_storage_credentials}')}
drive_type = business
auth_url = https://login.microsoftonline.com/<tenant id>/oauth2/v2.0/authorize
token_url = https://login.microsoftonline.com/<tenant id>/oauth2/v2.0/token

 
[one_drive_source]
type = onedrive
token = ${_credentialToken('${ops_ue_cloud_dt_source_credentials}')}
drive_id = ${_credentialUser('${ops_ue_cloud_dt_source_credentials}')}
drive_type = business
 
[one_drive_target]
type = onedrive
token = ${_credentialToken('${ops_ue_cloud_dt_target_credentials}')}
drive_id = ${_credentialUser('${ops_ue_cloud_dt_target_credentials}')}
drive_type = business
 
[linux_source]
type = local
 
[linux_target]
type = local
 
[windows_source]
type = local
 
[windows_target]
type = local

Input Fields

Fields Dependencies

This Universal Extension support Actions which can be categorized as follows:

Fields Description

The input fields for this Universal Extension are described below.

Task Examples

Configuration File Script

Transfer between Cloud Storages and Refresh Storage Credentials

List Cloud Storage

Synchronize local file system to Cloud Storage

Task Output

Output Only Fields

The output fields for this Universal Extension are described below.

Exit Codes

The exit codes for this Universal Extension are described below.

Extension Output

In the context of a workflow, subsequent tasks can rely on the information provided by this integration as Extension Output.

Attribute changed is populated as follows:

The Extension output contains the attribute result. Attribute result contains the following sub-attributes:

An example of the Extension Output for a "Copy Objects" Action is presented below.

{
    "exit_code": 0,
    "status_description": "Task executed successfully",
    "invocation": {
        "extension": "ue-cloud-dt",
        "version": "3.0.0",
        "fields": {
            "action": "Copy Objects",
            "config": "rclone.conf",
            "log_level": "DEBUG",
            "log_format": "JSON",
            "storage_system": null,
            "storage_type": "local",
            "storage_credentials": null,
            "source_storage": "ue_aws_s3_source",
            "source_credentials": {
                "user": "ue-aws-s3",
                "password": "****",
                "token": null,
                "key_location": null,
                "passphrase": null
            },
            "target_storage": "one_drive_target",
            "target_credentials": {
                "user": "rclone_one_drive",
                "password": "****",
                "token": null,
                "key_location": null,
                "passphrase": null
            },
            "controller_credentials": null,
            "filepath": null,
            "source_filepath": "john-bucket",
            "target_filepath": "john-drive",
            "use_filter": "Include", 
            "filter": "*.py",
            "additional_options": null,
            "overwrite_options": "Do Not Overwrite",
            "max_depth": 1,
            "error_on_no_transfer": false,
            "dry_run": true,
            "update_credentials": false,
            "controller_url": null,
            "refresh_storage_credentials": null,
            "storage_credentials_name": null,
            "source_credentials_name": "aws_se",
            "target_credentials_name": "one_drive",
            "controller_credentials_name": null,
            "configuration_file_name": "rclone.conf"
        }
    },
    "changed": true,
    "result": {
        "cmd": "rclone --config=//var/opt/universal/tmp/b4a333c4-497e-425c-8ce3-f8c5bdad9150.con --log-level=DEBUG --use-json-log --max-depth=1 --dry-run --progress --no-traverse --ignore-existing copy ue_aws_s3_source:john-bucket one_drive_target:john-drive --include *.py",
        "rc": 0
    }
}

STDOUT and STDERR

STDOUT/STDERR of Rclone is redirected to Task STDOUT/STDERR respectively and propagated during task execution. This is particularly useful for long-lasting data transfers. Retrieve Output while Task Instance is Running and press Refresh to get the latest STDOUT/STDERR.

Observability

Metrics are generated and exported by this integration, contributing to the UAC data observability, delivered since Universal Automation Center 7.5.0.0.

Metric: ue.cdt.rclone.speed

Metric: ue.cdt.rclone.duration

Metric: ue.cdt.rclone.transfers

Metric Attributes

The following attributes are applicable for ue.cdt.rclone.speed, ue.cdt.rclone.duration and ue.cdt.rclone.transfers.

Other Observability Configuration Options

Administrators are able to update the Universal Event Template to control additional attributes that can be sent out for Universal Event-based Metrics. However, this configuration should be approached with caution. When the possible distinct values of those attributes are high, that might lead to high cardinality issues.

Administrators can activate them with caution as follows:

1. Go to Administration → Universal Templates

2. Select the Universal Template

3. Click “Event Templates” Tab

4. Select the Name of the Event Template you want to update, which represents the metric you are interested in.

5. Update the “Optional Metric Labels” List the required Metric Labels.

How To

Import Grafana Dashboard

Users can benefit from a ready-to-use sample dashboard that this downloadable integration offers when observability features are used. It is located under the /observability/grafana/ directory inside the downloadable zip file from Stonebranch Integration Hub. Administrators should refer to the official Grafana User Guide on how to import a Grafana Dashboard.

Integration Modifications

Modifications applied by users or customers, before or after import, might affect the supportability of this integration. The following modifications are discouraged to retain the support level as applied for this integration.

• Python code modifications should not be done.

• Template Modifications

  • General Section

    • "Name", "Extension", "Variable Prefix", and "Icon" should not be changed.

  • Universal Template Details Section

    • "Template Type", "Agent Type", "Send Extension Variables", and "Always Cancel on Force Finish" should not be changed.

  • Result Processing Defaults Section

    • Success and Failure Exit codes should not be changed.

    • Success and Failure Output processing should not be changed.

  • Fields Restriction Section
    The setup of the template does not impose any restrictions, However with respect to the "Exit Code Processing Fields" section.

    1. Success/Failure exit codes need to be respected.

    2. In principle, as STDERR and STDOUT outputs can change in follow-up releases of this integration, they should not be considered as a reliable source for determining the success or failure of a task.

Event Template configuration related to “Metric Label Attributes” & “Optional Metric Labels” is allowed. However, administrators should be cautious of high cardinality scenarios that might occur

Users and customers are encouraged to report defects, or feature requests at Stonebranch Support Desk.

Document References

This document references the following documents.

Changelog

ue-cloud-dt-3.0.0 (2023-10-09)

Enhancements

• Changed: The progress bar, available on Universal Controllers of version 7.3 and above, now syncs with RClone's transfer completion (#34332).

• Changed: Improved clarity of RClone's log messages with respect to log level (#34332).

• Added: Introduced Observability metrics enabling enhanced visibility and understanding of behavior for Task Instances. This feature is enabled on Universal Controllers of version 7.5 and above (#34330).

• Added: Filter options are available for all list actions (#34455).

Deprecations and Breaking Changes

• Breaking Change: Updated the status descriptions to be more informative (#34335).

  Tasks or workflows evaluating the "Status Description" of the task Instance, either programmatically or within UAC, might be affected by it. In that case, they need to conform to the new "Status Description" Text

• Breaking Change: Extension Invocation Fields > Some fields are renamed for better readability (#34335).

• Breaking Change: Extension Invocation Fields > Credential fields are displayed as an object for better readability (#34335).

Fixes

• Fixed: RClone's command will no longer include duplicate flags (#34265).

Other

• Extra: The extension is supplemented with a Grafana dashboard that, by utilizing the produced observability metrics, calculates and displays statistical data (#34338).

ue-cloud-dt-2.0.1 (2023-07-28)

Fixes

• Fixed: Retrieving STDOUT/STDERR might block a Task Instance in Running status (#33732).

ue-cloud-dt-2.0.0 (2022-10-26)

Deprecations and Breaking Changes

• Breaking Change: Universal Template Name rename.
  This version of Universal Extension will have no effect on existing tasks created with Universal Template Cloud Data Transfer and extension archive ue-cloud-ft-1.0.0. Future updates on this integration will take place on top of ue-cloud-dt-2.0.0.

ue-cloud-ft-1.0.0 (2022-09-02)

Enhancements

• Added: Initial version (#28787)