Versions Compared

Old Version 43

changes.mady.by.user Stonebranch (Deactivated)

Saved on

compared with

New Version 44

changes.mady.by.user Kostas Papageorgiou

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: fixed internal links


Panel
Table of Contents

...

Template Name

Extension Name

Version

Status

Cloud Data Transfer

ue-cloud-dt

4 (Current 4.0.1)

Fixes and new features are introduced.

Cloud Data Transferue-cloud-dt3Hot Fixes Only (Until UAC 7.3 is End of Support)
Cloud Data Transferue-cloud-dt1 & 2 End of Support

Refer to changelogChangelog for version history information.

Warning

Version 4.0.0 is a major release and introduces breaking changes that might affect some users depending on their setup. Administrators are strongly advised to refer to Changelog for more information on the changes introduced in this release. Refer also to Migration Guide to apply changes in existing task definitions.  

Overview

This integration provides the capability to perform data transfers between cloud-based storage services and local or distributed file systems. It also provides data storage management capabilities like listing, creating, or deleting data storage objects. 


Key Features

This integration is equipped with the following key features.

...

NameTypeDescriptionVersion Information
ActionChoiceAction to be performed.

Available actions:
  • Copy Objects (default)
  • Copy And Rename
  • Copy From URL
  • Move Objects
  • Move And Rename
  • List Objects
  • List Directories
  • List Objects With Details
  • List Objects In JSON Format
  • Create Object
  • Delete Object
  • Delete Empty Directory
  • Synchronize
Introduced in 1.0.0
Configuration FileScript

This integration depends on Rclone. This field models the Rclone Configuration file, which contains all required parameters to connect to the Storage System, Source Storage and Target Storage. The filename should have the ".conf" suffix.

Example: rclone.conf

Introduced in 1.0.0
Storage SystemDynamic Choice FieldRemote or Local storage to execute the selected Action.
Execute the Dynamic Choice Field and retrieve all the available Storages from Configuration File.

Required when one of the Single storage actions is selected.		Introduced in 1.0.0
Source StorageDynamic Choice FieldRemote or Local source storage, where transfer Action is initiated.
Execute the Dynamic Choice Field and retrieve automatically all the available Storages from Configuration File.

Required when one of the Two storage actions is selected.		Introduced in 1.0.0
Target StorageDynamic Choice FieldRemote or Local target storage, where transferred data is stored.
Execute the Dynamic Choice Field and retrieve automatically all the available Storages from Configuration File.

Required when one of the Two storage actions is selected.		Introduced in 1.0.0
Storage CredentialsCredential

Credentials needed to connect to the selected Storage System.
For Storage System of type local, the standard Agent's Credentials field should be provided.

From Universal Controller 7.6 onwards a variable that holds the Credential Name can be used when related checkbox is selected. In this case the content of this field should be ${variable_name}


Required when one of the Single storage actions is selected.

Introduced in 1.0.0
Source CredentialsCredential

Credentials needed to connect to the selected Source Storage.

From Universal Controller 7.6 onwards a variable that holds the Credential Name can be used when related checkbox is selected. In this case the content of this field should be ${variable_name}

Required when one of the Two storage actions is selected. For Storage System of type local, the standard Agent's Credentials field should be provided.

Introduced in 1.0.0
Target CredentialsCredential

Credentials needed to connect to the selected Target Storage.
From Universal Controller 7.6 onwards a variable that holds the Credential Name can be used when related checkbox is selected. In this case the content of this field should be ${variable_name}


Required when one of the Two storage actions is selected.

For Storage System of type local, the standard Agent's Credentials field should be provided.

Introduced in 1.0.0
FilepathTextFile or directory path from where to retrieve data for the selected Action.

Required when one of the Single storage actions is selected.		Introduced in 1.0.0
Source FilepathTextFile or directory path from where to retrieve data for the selected Action.

Required when one of the Two storage actions is selected.		Introduced in 1.0.0
Target FilepathTextFile or directory path where transferred data will be stored for the selected Action.

Required when one of the Two storage actions is selected.		Introduced in 1.0.0
Update CredentialsCheckbox 

If the remote storage uses OAuth for its authorization, Rclone can be adjusted to refresh the respective token in its Configuration File during execution time. Enabling this field, will ensure that the runtime-updated token from Rclone, will get provisioned to the respective Credential Field on the Universal Controller.

This field is used in conjunction with fields Controller URL, Controller Credentials, and optionally the Refresh Storage Credentials. See the respective examples in Refresh Storage Credentials chapter.


The default setting is unchecked.

Introduced in 1.0.0
Controller URLTextController URL where Storage Credentials, Source Credentials, Target Credentials are stored.

Required when Update Credentials is checked.		Introduced in 1.0.0
Controller CredentialsCredentialsController user's credentials, used for logging and updating Storage Credentials, Source Credentials, Target Credentials. The Credentials definition should be as follows:
  • User as "Runtime User".
  • User Password as "Runtime Password".
Note: The provided user should have sufficient Credential permissions.

Required when Update Credentials is checked.		Introduced in 1.0.0

Refresh Storage Credentials

Anchor
Refresh Storage Credential
Refresh Storage Credential

Array

Note: This Array is not required to be configured in case the Storage type is "One Drive".

This Array field is available when Update Credentials is checked. Is used as a mapping table between the Configuration File locator that has been updated by Rclone during execution, and its corresponding Universal Controller Credential field.

Populate this array with the following format:

  • Configuration File Details: "<Storage/Source/Target Storage>"."<Configuration File's field to be refreshed by Rclone>"
  • Storage Credentials Details: "<Storage/Source/Target Credentials name>"."<Storage/Source/Target Credentials field to be updated>"

The result of this action is that the Configuration File's locator value, is copied to the provided Storage Credential's field. Each line of this Array is evaluated, and related mapping is performed. See the Refresh authorization fields for other remotes.

Note: Refresh of Storage/Source/Target Credentials is performed via Controller's REST API. See Network 150079703 Requirements .

Introduced in 1.0.0
Use FilterChoiceThe filter type that is applied on the Action.

Available options:
  • -- None – (default)
  • Include
  • Exclude

Optional for all, but Create Object Action.		Introduced in 1.0.0
FilterTextFilter as regular expression that is applied on the Action.

Required when Use Filter is checked.		Introduced in 1.0.0
Overwrite OptionsChoiceOptions for overwriting files in the Target Storage.

Available options:
  • Overwrite
  • Do Not Overwrite (default)
  • Append Timestamp In Filename

Optional when one of the Two storage actions is selected.		Introduced in 1.0.0
Recursion DepthIntegerRecursion depth that is applied. Default value 1 means that no recursion will be applied.

Note: Use this option with caution when one of the following Actions are selected, as this might result in doing actions on unnecessary files/cloud objects. For more information on Recursion Depth, you can refer to the official Rclone documentation:
  • Copy Objects
  • Copy And Rename
  • Move Objects
  • Move And Rename
  • Delete Object
  • Delete Empty Directory
  • Synchronize


The default value is 1.

Introduced in 1.0.0
Error On No File TransferCheckbox

When enabled and when no files are transferred a failure exit code with value equal to 21 is raised.

Optional when one of the Two storage actions is selected.


The default setting is unchecked.

Introduced in 1.0.0
Dry-RunCheckboxWhen enabled, performs a trial run with no permanent changes of the selected Action.

Note: It is recommended to execute a Dry-Run as a test prior to the final task configuration when the following Actions are used:
  • Copy Objects
  • Copy And Rename
  • Move Objects
  • Move And Rename
  • Delete Object
  • Delete Empty Directory
  • Synchronize


The default value is unchecked.

Introduced in 1.0.0
Additional OptionsTextSpace-separated Rclone options applied to the selected Action.Introduced in 1.0.0
Log FormatChoiceOption for STDERR logging format.

Available options:
  • Text (default)
  • JSON
Introduced in 1.0.0

...

Modifications of this integration, applied by users or customers, before or after import, might affect the supportability of this integration. For more information refer to Integration Modifications.

Configure Universal Task

Anchor
configure_universal_task
configure_universal_task

To configure a new Universal Task, there are three steps required:

  • Create required Resolvable Credentials. Required as Input Fields on the Universal Task Configuration.
  • Create a new Script of type Data, for the Configuration File and populate it according to the following section of Setup Rclone Configuration File as UAC Script.
  • Create a new task, and enter the task-specific details that were created in the Universal Template.

...

Note

This integration should work properly against other Storage Systems as well, as long as the integration interface (see chapter Configure Universal Task) and functionalities listed in this document fulfill the needs of the Storage System.

For functionalities required for specific Storage Systems, users and customers are encouraged to open a Feature Request in our Customer Support Portal.

...

SystemRclone Configuration Documentation LinkExample
AWS S3AWS S3 Rclone Configuration Guide


Code Block
languagetext
titleExample with Credentials as UAC Functions
collapsetrue
[aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_storage_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_storage_credentials}')}
region = us-east-2

[aws_s3_source]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_source_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_source_credentials}')}
region = us-east-2
acl = bucket-owner-full-control
role_arn = arn:aws:iam::<account_id>:role/<role policy>

[aws_s3_target]
type = s3
provider = AWS
env_auth = false
access_key_id = ${_credentialUser('${ops_ue_cloud_target_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_target_credentials}')}
region = us-east-2


Google Cloud Storage (Google Drive)Google Drive Rclone Configuration Guide


Code Block
languagetext
titleExample with Credentials as UAC Functions
collapsetrue
[google_cloud_storage]
type = google cloud storage
service_account_file = ${_credentialPwd('${ops_ue_cloud_storage_credentials}')}
object_acl = bucketOwnerFullControl
project_number = johnprojectno
location = europe-west3

[google_cloud_storage_source]
type = google cloud storage
service_account_file = ${_credentialPwd('${ops_ue_cloud_source_credentials}')}
object_acl = bucketOwnerFullControl
project_number = johnprojectno
location = europe-west3

[google_cloud_storage_target]
type = google cloud storage
service_account_file = ${_credentialPwd('${ops_ue_cloud_target_credentials}')}
object_acl = bucketOwnerFullControl
project_number = johnprojectno
location = europe-west3


Microsoft OneDrive/Sharepoint 

Microsoft OneDrive/Sharepoint Configuration Guide

For One Drive / Sharepoint data transfers, refer also to Refresh Storage Credentials section.


Code Block
languagetext
titleExample (OneDrive) with Credentials as UAC Functions
collapsetrue
[one_drive]
type = onedrive
client_id = ${_credentialUser('${ops_ue_cloud_storage_credentials}')}
client_secret = ${_credentialPassword('${ops_ue_cloud_storage_credentials}')}
drive_id = ${_credentialUser('${ops_ue_cloud_storage_credentials}')}
drive_type = business
token = ${_credentialToken('${ops_ue_cloud_storage_credentials}')}

 
[one_drive_source]
type = onedrive
token = ${_credentialToken('${ops_ue_cloud_source_credentials}')}
client_id = ${_credentialUser('${ops_ue_cloud_source_credentials}')}
client_secret = ${_credentialPassword('${ops_ue_cloud_source_credentials}')}
drive_id = ${_credentialUser('${ops_ue_cloud_source_credentials}')}
drive_type = business
token = ${_credentialToken('${ops_ue_cloud_storage_credentials}')}
 
[one_drive_target]
type = onedrive
client_id = ${_credentialUser('${ops_ue_cloud_target_credentials}')}
client_secret = ${_credentialPassword('${ops_ue_cloud_target_credentials}')}
drive_type = business
token = ${_credentialToken('${ops_ue_cloud_target_storage_credentials}')}


Local Filesystem 

Local Filesystem Configuration Guide


Code Block
languagetext
titleExample
collapsetrue
[linux_source]
type = local
 
[linux_target]
type = local
 
[windows_source]
type = local
 
[windows_target]
type = local


...

There are several ways that Credential Functions can be used. To make the Configuration File immune to changes of credential names used in the task definition it is recommended to follow the Configuration through Credential Field Variables approach. Another benefit of this approach is that it is compatible with all the Universal Controller versions supported by this integration. However, alternative approaches are Configuration through Credential Name and Configuration through Credential as Variable, which rely on changes performed on the Configuration File itself, rather on the task definition. The next chapters provide more information on them. 

...

The task input fields related to this configuration are Update Credentials, Controller URL, Controller Credentials, and Refresh Storage Credentials. 


An example of such a configuration can be found below:

...

When upgrading major versions, the following actions should be performed to migrate properly task definitions due to the breaking changes:

  • Tasks which reference Cloud Data Transfer task instance variables (for example sibling tasks within a Workflow), should be updated to reference the Cloud Data Transfer variables as ${ops_ue_cloud_<field_name>}.
  • Configuration File should be updated when credential fields are referenced:

...