/
Inter-Cloud Data Transfer

Inter-Cloud Data Transfer

Disclaimer

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at Stonebranch Integration Hub - Terms of Use.


Version Information

Template Name

Extension Name

Version

Status

Cloud Data Transfer

ue-cloud-dt

4 (Current 4.0.2)

Fixes and new features are introduced.

Cloud Data Transferue-cloud-dt3Hot Fixes Only (Until UAC 7.3 is End of Support)
Cloud Data Transferue-cloud-dt1 & 2 End of Support

Refer to Changelog for version history information.

Version 4.0.0 is a major release and introduces breaking changes that might affect some users depending on their setup. Administrators are strongly advised to refer to Changelog for more information on the changes introduced in this release. Refer also to Migration Guide to apply changes in existing task definitions.  

Overview

This integration provides the capability to perform data transfers between cloud-based storage services and local or distributed file systems. It also provides data storage management capabilities like listing, creating, or deleting data storage objects. 


Key Features

This integration is equipped with the following key features.


FeatureDescription
Data management and synchronization
  • Copy, move, and synchronize data between two storages.
  • Copy a URL's content to a cloud or local destination without saving it in temporary storage.
  • List data on a storage service, including listing with details or in JSON format for machine parsing.
  • Create objects on a storage.
  • Delete objects from a storage.
Efficient and secure data transfers
  • Fast transfers for objects stored in the same region.
  • Preserve timestamps and checksums verification.
  • Transfers can optionally support encryption, caching, compression, and chunking.
  • Dynamic token refresh for OneDrive cloud storage, observing the OAuth2 authorization and sign-in flow.
  • Support for dry runs. Allows users to execute a Universal Task without making any permanent changes to the target storage.
  • Advanced filtering capability for files or objects to be listed or transferred.
Complementary capabilities
  • Additional customized options.
  • Option to mark the Universal Task as Failed when no files have been transferred.
  • Progress of the selected Action is visible, during Universal Task Instance execution.
  • Text or JSON formatted output.
Observability
  • Rich observability capabilities (starting from UAC 7.5.0.0)


Requirements

This integration requires a Universal Agent, a Python runtime to execute the Universal Task.

Area

Details

Python Version

Requires Python 3.7 or 3.11.  Tested with Python 3.7.6 and Python 3.11.6.

Universal Agent Compatibility

  • Compatible with Universal Agent for Windows x64 and version >= 7.4.0.0.

  • Compatible with Universal Agent for Linux and version >= 7.4.0.0.

Universal Controller Compatibility

Universal Controller Version >= 7.4.0.0.

Network and Connectivity

Universal Agent should be able to establish a connection with:

  • The remote storage providers used at the Universal Task Definition.
  • The Universal Controller host. This is required only in case Refresh Storage Credentials field is used



Supported Actions

This integration supports multiple actions which can be grouped into the following categories: 

CategoryActionsRequired Input Fields
Single storage actions
  • List Objects
  • List Directories
  • List Objects With Details
  • List Objects In JSON Format
  • Create Object
  • Delete Object
  • Delete Empty Directory
  • Storage System
  • Storage Credentials
  • Filepath
Two storage actions
  • Copy Objects
  • Copy And Rename
  • Copy From URL
  • Move Objects
  • Move And Rename
  • Synchronize
  • Source Storage
  • Source Credentials
  • Source Filepath
  • Target Storage
  • Target Credentials
  • Target Filepath

Action Type: Single Storage Action

Single storage actions involve listing and managing existing data objects in a local or remote storage. 


Configuration Examples


User Scenario:

List Directories action, using a filter and with output log in Text format.

User Scenario:

Create Object action and with output as JSON format.


Action Output


Output Type

Description

Examples

EXTENSION

The extension output provides the following information:

  • exit_code,  status_description: General info regarding the task execution. For more information, users can refer to the Exit Codes section.

  • invocation.fields: The task configuration used for this task execution.

  • result.cmd: The executed command line.

  • result.rc: The return code from the executed command line.

Successful Execution
{
	"exit_code": 0,
	"status_description": "Task executed successfully",
	"invocation": {
		"extension": "ue-cloud-dt",
		"version": "4.0.0",
		"fields": {
			"action": "List Directories",
             ...
		}
	},
	"changed": false,
	"result": {
		"cmd": "rclone lsd ue_aws_s3:john-drive/--config=C:\\DATA\\tmp\\54cd05b2-1528-46d2-8f9c- 8bc09dfdbf4.conf --log-level=DEBUG --max-depth=4 --use-json-log --exclude ue_execution/compiled",
		"rc": 0
	}
}
Failed Execution
{
&nb