Disclaimer

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at https://www.stonebranch.com/integration-hub/Terms-and-Privacy/Terms-of-Use/

Overview

This Universal Extension is an interface to Rclone that provides the capability to monitor files/directories from across different cloud storages, as well as local or distributed file systems. Additionally and upon successful monitor, this extension publishes Local Universal Events. To properly handle these events and take action on them, this Universal Extension optionally can be attached as a Publisher to a Universal Monitor Task (see more details in section 150079617).
Rclone, is the open source command line program, that is utilized to accomplish all the actions supported in this Extension.

Version Information

Refer to Changelog for version history information.

Software Requirements

This integration requires a Universal Agent, a Python runtime to execute the Universal Task, and a Rclone executable.

Software Requirements for Universal Template and Universal Task

Requires Python 3.7.0 or higher. Tested with the Universal Agent bundled Python distribution.

Software Requirements for Universal Agent

Both Windows and Linux agents are supported.

• Universal Agent for Windows x64 Version 7.2.0.0 and later with python options installed.
• Universal Agent for Linux Version 7.2.0.0 and later with python options installed.

Software Requirements for Universal Controller

Universal Controller Version 7.2.0.0 and later.

Software Requirements for Rclone

Rclone needs to be installed on the same server where the Universal Agent is installed. Rclone binary should be stored on a location where Universal Agent has access and permissions to execute it.

Network and Connectivity Requirements

Universal Agent should be able to establish connection with:

• The remote storages that are provided as input for the Universal Task.
• The Universal Controller host. This is required only in case Refresh Storage Credentials field is used.

Key Features

This Universal Extension supports the following key features:

• Actions
  • Monitor objects creation
  • Monitor objects change - change based on modification time
  • Monitor objects deletion
• Options
  • Trigger On Existence - available for Action Monitor On Create.
  • Advanced filtering capability for objects to be monitored.
  • Support for providing additional Rclone options according to the user needs.
• Universal Events
  • Event published upon single object creation, holding information about the new object.
  • Event published upon single object change, holding information about the updated file including the latest object size and modification time.
  • Event published upon single object deletion, holding information about the deleted object.
  • Event published upon single object existence, holding information about the object.

Integrated Storage Systems

Rclone is integrated with multiple storage systems. This Universal Extension has been tested against the following Storage Systems.

• Amazon S3
• Google Cloud Storage
• Microsoft OneDrive Business
• Local file system (Linux, Windows)

Import Universal Template

To use the Universal Template, you first must perform the following steps.

1. This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.

2. To import the Universal Template into your Controller, follow these instructions.

3. When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list.

Modifications of this integration, applied by users or customers, before or after import, might affect the supportability of this integration. For more information refer to Integration Modifications.

Configure Universal Task

To configure a new Universal Task, there are three steps required:

• Create required Resolvable Credentials. Required as Input Fields on the Universal Task Configuration.
• Create a new Script of type Data, for the Configuration File and populate it according to the following section of Setup Rclone Configuration File. Required as Input Fields on the Universal Task Configuration.
• Create a new task, and enter the task-specific details that were created in the Universal Template.

Setup Rclone Configuration File

1. The configuration file is following the INI format, and contains all required parameters and credentials to connect to a Storage System. All configuration options per Storage System can be found in the respective "Config Rclone documentation" .

2. It is advised that account credentials, tokens, or any other essential information to be passed as an encrypted Resolvable Credential field. To set a value with the respective Credential field, please make sure to use the appropriate Credential function inside the configuration file.

   'Runtime Password' Credentials field supports values with up to 512 characters.

   Hard coded values can be provided in the script file, as well. However, it is not advised for security reasons.

3. It is recommended to configure a storage system section, one to one with the Credential fields in this Universal Extension (please refer to the Input Fields section below).

4. For local file system Storages, no credentials are required in the configuration file. The selected action will be executed with the Credentials that are provided in the "Agent Details" of the Universal Task. If no Credentials are provided for the used Agent, then the same user that runs ubroker daemon will be used.

5. Script Type should be Data, and option Resolve UAC Variables should be enabled.

The indicative Configuration File below, contains the basic connection parameters (flags):

[ue_aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_dm_storage_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_dm_storage_credentials}')}
region = us-east-2


[google_cloud_storage]
type = google cloud storage
service_account_file = ${_credentialPwd('${ops_ue_cloud_dm_storage_credentials}')}
object_acl = bucketOwnerFullControl
project_number = johnprojectno
location = europe-west3

[one_drive]
type = onedrive
token = ${_credentialToken('${ops_ue_cloud_dm_storage_credentials}')}
drive_id = ${_credentialUser('${ops_ue_cloud_dm_storage_credentials}')}
drive_type = business
auth_url = https://login.microsoftonline.com/<tenant id>/oauth2/v2.0/authorize
token_url = https://login.microsoftonline.com/<tenant id>/oauth2/v2.0/token

[local_filesystem]
type = local

Input Fields

The input fields for this Universal Extension are described below.

Cloud Data Monitor Events

This Universal Extension publishes Universal Events when objects are monitored succesfully, based on the selected Action. The suggested implementation to accomplish this behavior is to associate a Cloud Data Monitor Task(event publisher) with a Universal Monitor Task(event consumer).

The Universal Events for this Universal Extension are described below.

• Event Monitor On Create

  Attribute	Description	Type
  Storage System	The Storage System where the monitored object is stored. This is the input field Storage System.	Text
  Storage Filepath	The Storage System where the monitored object is stored. This is the input field Storage Filepath.	Text
  Filename	The name of the monitored object.	Text
  Correlation ID	Custom information field. This is the input field Correlation ID.	Text

• Event Monitor On Change

  Attribute	Description	Type
  Storage System	The Storage System where the monitored object is stored. This is the input field Storage System.	Text
  Storage Filepath	The Storage System where the monitored object is stored. This is the input field Storage Filepath.	Text
  Filename	The name of the monitored object.	Text
  Filesize	The updated size of the monitored object.	Text
  File Modified Time	The time when the object last changed.	Text
  Correlation ID	Custom information field. This is the input field Correlation ID.	Text

• Event Monitor On Delete

  Attribute	Description	Type
  Storage System	The Storage System where the monitored object is stored. This is the input field Storage System.	Text
  Storage Filepath	The Storage System where the monitored object is stored. This is the input field Storage Filepath.	Text
  Filename	The name of the monitored object.	Text
  Correlation ID	Custom information field. This is the input field Correlation ID.</