/
Inter-Cloud Data Monitor

Inter-Cloud Data Monitor

Disclaimer

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at Stonebranch Integration Hub - Terms of Use.

Version Information

Template Name

Extension Name

Version

Status

Template Name

Extension Name

Version

Status

Cloud Data Monitor

ue-cloud-dm

2 (Current 2.5.0)

Fixes and new features are introduced.

Cloud Data Monitor

ue-cloud-dm

1

Hot Fixes Only (Until UAC 7.5 is End of Support)

Refer to Changelog for version history information.

Version 2.0.0 is a major release and introduces breaking changes that might affect some users depending on their setup. Administrators are strongly advised to refer to Changelog for more information on the changes introduced in this release. Refer also to the Migration Guide to apply changes in existing task definitions.  

Overview

One of the key use cases for Service Orchestration and Automation Platforms is the ability to detect changes in files or directories and respond to these events by executing specific actions.

This integration enables monitoring of files and directories across various local and cloud storage systems. When a change is detected, Local Universal Events are triggered and can be processed within Universal Controller, such as initiating a task or workflow. 

Key Features

This Universal Extension supports the following key features:

Feature

Description

Feature

Description

Monitoring

  • Monitor one or multiple objects' creation, modification, or deletion.

  • Support for multiple Storage Systems across multiple platforms.

Options

  • Event-driven task execution when combined with Universal Triggers.

  • Advanced filtering capabilities.

  • Dynamic token refresh for OneDrive cloud storage.

Universal Events

  • Universal Event is published upon occurring action, sharing information about the object that prompted it.

Rclone

  • This integration is bundled with Rclone v1.73.2. Installing Rclone binaries on the Agent is not required.

Requirements

This integration requires a Universal Agent, and a Python runtime to execute the Universal Task. The following are the requirements for Version 2

Area

Details

Area

Details

Python Version

Requires Python 3.11.  Tested with Python 3.11.6.

Universal Agent Compatibility

  • Compatible with Universal Agent for Windows x64 and version >= 7.6.0.0.

  • Compatible with Universal Agent for Linux and version >= 7.6.0.0.

Universal Controller Compatibility

Universal Controller Version >= 7.6.0.0.

Network and Connectivity

Universal Agent should be able to establish a connection with:

  • The remote storage providers used at the Universal Task Definition.

  • The Universal Controller host. This is required only in case Update Credentials field is used.

Supported Actions

This integration supports the following actions.

Action: Monitor On Create

This action enables monitoring for the creation of one or multiple objects. The object can either be a file or a directory. Selecting this action enables the Inter-Cloud Data Monitor#Trigger On Existence option, which treats the initial files of the directory as a file creation event. If this option is enabled, the user has to set a filter based on rclone filtering rules.

Configuration examples

User Scenario: Monitor On Create minimum configuration.

 

User Scenario: Monitor On Create with the Inter-Cloud Data Monitor#Trigger On Existence option enabled. The Filter option is used to select specific files.

Action Output

This extension provides EXTENSION output on Cancel or on Failure.

OutputType

Description

Examples

OutputType

Description

Examples

EXTENSION

The EXTENSION OUTPUT provides the following information:

  • exit_code, status_description: General info regarding the task execution. For more information, refer to the Exit Codes section.

  • invocation.fields: The task configuration used for this task execution.

  • result.cmd: The executed command line.

  • result.rc: The return code from the executed command line.

  • result.errors: List of errors that might have occurred during Universal Task Execution.

Successful Execution
{ "exit_code": 0, "status_description": "Task executed successfully", "invocation": { "extension": "ue-cloud-dm", "version": "2.1.0", "fields": { "action": "Monitor On Create", "trigger_on_existence": false, "configuration_file": "/home/user/data/tmp/6e9d8fd1-167c-4e74-b210-898d1eaaebef.con", "storage_system": "local_ua", "storage_filepath": "/home/ue-dev/USERS/user/cloud-dm/", "files_only": false, "storage_credentials": { "user": "599910b6-1234-1234-abcdef-1234cc63bd0", "password": "****", "token": "****", "key_location": null, "passphrase": "****" }, "update_credentials": false, "controller_url": null, "controller_credentials": null, "refresh_storage_credentials": null, "monitor_time_interval": 10, "use_storage_modtime": false, "recursion_depth": 1, "use_filter": "Include", "filter": "{{report.*}}", "additional_options": null, "tag": null, "correlation_id": null, "log_level": "DEBUG", "configuration_file_name": "rclone.conf", "storage_credentials_user": null, "storage_credentials_password": null, "storage_credentials_token": null, "storage_credentials_name": "local_credentials", "controller_credentials_user": null, "controller_credentials_password": null, "controller_credentials_name": null } }, "result": { "cmd": "/home/user/data/uag/extensions/.ue-cloud-dm/vendor/rclone lsjson local_ua:/home/ue-dev/USERS/user/cloud-dm/ --config=/home/user/data/tmp/2a100d7f-08b5-43c2-b20d-344a2c2d85a9.con --max-depth=1 --log-level=DEBUG" --include {{report.*}}", "rc": 0 } }
Failed Execution
{ "exit_code": 1, "status_description": "Execution Failed: See STDERR for more information.", "invocation": { "extension": "ue-cloud-dm", "version": "2.1.0", "fields": { "update_credentials": false, "configuration_file": "/home/user/data/tmp/2a100d7f-08b5-43c2-b20d-344a2c2d85a9.con", "refresh_storage_credentials": null, "recursion_depth": 1, "tag": "extensions_test", "monitor_time_interval": 300, "use_filter": "-- None --", "additional_options": null, "files_only": false, "controller_url": null, "storage_system": "one_drive", "use_storage_modtime": false, "trigger_on_existence": true, "action": "Monitor On Create", "filter": null, "storage_filepath": "/user-storage/", "log_level": "DEBUG", "storage_credentials": { "user": "599910b6-1234-1234-abcdef-1234cc63bd0", "password": "****", "key_location": null, "passphrase": "****", "token": "****" } } }, "result": { "cmd": "/home/user/data/uag/extensions/.ue-cloud-dm/vendor/rclone lsjson one_drive:/user-storage/ --config=/home/user/data/tmp/2a100d7f-08b5-43c2-20d-344a2c2d85a9.con --max-depth=1 --log-level=DEBUG", "rc": 1 "errors": [ "Failed to create file system for \"one_drive:/user-storage/\": failed to get root: Get \"https://graph.microsoft.com/v1.0/drives/HASH/root\": couldn't fetch token: invalid_client: if you're using your own client id/secret, make sure they're properly set up following the docs" ] } }

Action: Monitor On Change and Monitor On Delete

The Monitor On Change action monitors if the object's modification time has been updated, while the Monitor On Delete action monitors whether an object has been deleted. These two actions share the same configuration.

Configuration examples

User Scenario: Monitor On Change with a custom Event Tag and the Use System Storage Modified Time field selected.

 

User Scenario: Monitor On Delete using a Filter and the Files Only field. The Filter option is used to select specific files.

Action Output

This extension provides EXTENSION output on Cancel, or on Failure.

OutputType

Description

Examples

OutputType

Description

Examples

EXTENSION

The extension output provides the following information:

  • exit_code, status_description: General info regarding the task execution. For more information, refer to the Exit Codes section.

  • invocation.fields: The task configuration used for this task execution.

  • result.cmd: The executed command line.

  • result.rc: The return code from the executed command line.

  • result.errors: List of errors that might have occurred during Universal Task Execution.

Successful Execution
{ "exit_code": 0, "status_description": "Task executed successfully", "invocation": { "extension": "ue-cloud-dm", "version": "2.1.0", "fields": { "action": "Monitor On Change", "trigger_on_existence": false, "configuration_file": "/home/user/data/tmp/6e9d8fd1-167c-4e74-b210-898d1eaaebef.con", "storage_system": "local_ua", "storage_filepath": "/home/ue-dev/USERS/user/cloud-dm/", "files_only": false, "storage_credentials": { "user": "599910b6-1234-1234-abcdef-1234cc63bd0", "password": "****", "token": "****", "key_location": null, "passphrase": "****" }, "update_credentials": false, "controller_url": null, "controller_credentials": null, "refresh_storage_credentials": null, "monitor_time_interval": 10, "use_storage_modtime": false, "recursion_depth": 1, "use_filter": "Include", "filter": "{{report.*}}", "additional_options": null, "tag": null, "correlation_id": null, "log_level": "DEBUG", "configuration_file_name": "rclone.conf", "storage_credentials_user": null, "storage_credentials_password": null, "storage_credentials_token": null, "storage_credentials_name": "local_credentials", "controller_credentials_user": null, "controller_credentials_password": null, "controller_credentials_name": null } }, "result": { "cmd": "/home/user/data/uag/extensions/.ue-cloud-dm/vendor/rclone lsjson one_drive:/user-storage/ --config=/home/user/data/tmp/2a100d7f-08b5-43c2-20d-344a2c2d85a9.con --max-depth=1 --log-level=DEBUG", "rc": 0 } }
Failed Execution
{ "exit_code": 1, "status_description": ""Execution Failed: See STDERR for more information.", "invocation": { "extension": "ue-cloud-dm", "version": "2.1.0", "fields": { "update_credentials": true, "configuration_file": "/home/user/data/tmp/fa02639e-e7c4-4f26-90db-78a7b09acce8.con", "refresh_storage_credentials": null, "recursion_depth": 1, "tag": "extensions_test", "monitor_time_interval": 300, "use_filter": "-- None --", "additional_options": null, "files_only": false, "controller_url": "http://controller.stonebranch.org:8080/uc/", "storage_system": "one_drive", "use_storage_modtime": false, "trigger_on_existence": false, "action": "Monitor On Delete", "filter": null, "storage_filepath": "/invalid_directory/", "log_level": "DEBUG", "controller_credentials": { "user": "ops.****", "password": "****", "key_location": null, "passphrase": "****", "token": "****" }, "storage_credentials": { "user": "12342de-1234-1234-abcdef-1234cc6fff", "password": "****", "key_location": null, "passphrase": "****", "token": "****" } } }, "result": { "cmd": "/home/user/data/uag/extensions/.ue-cloud-dm/vendor/rclone lsjson one_drive:/invalid_directory/ --config=/home/user/data/tmp/fa02639e-e7c4-4f26-90db-78a7b09acce8.con --max-depth=1 --log-level=DEBUG", "rc": 3 "errors": [ "error listing: directory not found", "Failed to lsjson with 2 errors: last error was: error in ListJSON: directory not found" ] } }

Input Fields

The input fields for this Universal Extension are described below.

Field

Input type

Default value

Type

Description

Version Information

Field

Input type

Default value

Type

Description

Version Information

Action

Required

Monitor On Create

Choice

Action to be performed.

Available actions:

  • Monitor On Create

  • Monitor On Change

  • Monitor On Delete

This field is included as an Event Attribute.

Introduced in 1.0.0

Trigger On Existence

Optional

False

Boolean

When this flag is enabled and objects already exist on the given storage, an event will be published. Otherwise, the Universal Task will keep monitoring for new object creation.

Optional when Action "Monitor on Create" is selected.

Introduced in 1.0.0

Configuration File

Required

-

Script

This integration depends on Rclone. This field models the Rclone Configuration file, which contains all the required parameters to connect to the Storage System. The filename should have the ".conf" suffix.

Example: rclone.conf

Introduced in 1.0.0

Storage System

Required

-

Dynamic Choice Field

The remote or Local storage to execute the selected Action. The magnifier button next to the field can be used to retrieve all available Storage Systems from the Configuration File.

This field is included as an Event Attribute.

Introduced in 1.0.0

Storage Credentials

Optional

-

Credential

Credentials needed to connect to the selected Storage System.
For Storage System of type "local", the standard Agent's Credentials field should be provided.

 

From Universal Controller 7.6 onwards, a variable that holds the Credential Name can be used when the related checkbox is selected. In this case, the content of this field should be ${variable_name}

Introduced in 1.0.0

Storage Filepath

Optional

-

Text

The path to monitor objects for the selected Action.

This field is included as an Event Attribute.

Introduced in 1.0.0

Update Credentials

Optional

False

Boolean

If the remote storage uses OAuth for its authorization, Rclone can be adjusted to refresh the respective token in its Configuration File during execution time. Enabling this field will ensure that the runtime-updated token from Rclone will be provisioned to the respective Credential Field on the Universal Controller.

This field is used in conjunction with fields Controller URL, Controller Credentials, and optionally Refresh Storage Credentials. See the respective examples in the Refresh Storage Credentials chapter.

The default setting is unchecked.

Introduced in 1.0.0

SSL Certificate Verification

Optional

True

Boolean

When this flag is disabled, SSL Verification is skipped when connecting to Universal Controllers using HTTPS, and a CA Bundle is not required.

To provide a CA Bundle, consult the SSL Verification section.

 

Optional when Update Credentials is checked.

Introduced in 2.0.0

Controller URL

Optional

-

Text

Controller URL where Storage Credentials are stored.

Required when Update Credentials is checked.

Introduced in 1.0.0

Controller Credentials

Optional

-

Credentials

Controller user's credentials, used for logging and updating Storage Credentials. The Credentials definition should be as follows:

  • User as "Runtime User".

  • User Password as "Runtime Password".

Note: The user should have sufficient Credential permissions.

Required when Update Credentials is checked.

Introduced in 1.0.0

Refresh Storage Credentials

Optional

-

Array

Note: This Array is not required to be configured if the configuration (storage) is of type "box" or "onedrive". For more information, refer to the Automatic Refresh of Access Tokens of the Environment Variables section.

This Array field is available when Update Credentials is checked. It is used as a mapping table between the Configuration File locator that Rclone has updated during execution, and its corresponding Universal Controller Credential field.

Populate this array with the following format:

  • Configuration File Details: "<Storage System name>"."<Configuration File's field to be refreshed by Rclone>"

  • Storage Credentials Details: "<Storage Credentials name>"."<Storage Credentials field to be updated>"

This option copies the Configuration file's field value to the provided Storage Credential field. Each line of this Array is evaluated, and related mapping is performed. See Refresh Storage Credentials for other remotes.

 

Note: Refresh of Storage Credentials is performed via the Controller's REST API. See Network and Connectivity Requirements

Introduced in 1.0.0

Monitor Time Interval

Optional

300

Integer

The interval period in seconds used to monitor the given Storage Filepath.

Introduced in 1.0.0

Use Filter

Optional

-- None --

Choice

The filter type that is applied to the Action.

Available options:

  • -- None --

  • Include

  • Exclude

  • File

Introduced in 1.0.0

Filter

Optional

-

Text / Script

Filter based on Rclone filtering rules that is applied to the Action.  It is converted to a Script field when the "File" Use Filter option is selected. For more information on using Filter files, see Rclone Filtering.

Required when Use Filter is checked.

Introduced in 1.0.0

Files Only

Optional

False

Boolean

When enabled, directories will be ignored and only files will be monitored.

Introduced in 1.0.0

Use Storage System Modified Time

Optional

True

Boolean

When enabled, the modified time of the Storage System will be used for the object's modification time, rather than the original object's modified time metadata.

Introduced in 1.0.0

Recursion Depth

Optional

1

Integer

Recursion depth that is applied. Default value 1 means that no recursion will be applied.

Introduced in 1.0.0

Additional Options

Optional

-

Text

Space-separated Rclone options that applied to the selected Action.

Introduced in 1.0.0

Event Tag

Optional

-

Text

This field allows task authors to define a 'tag' attribute for the generated universal events, enabling more filtering capabilities at the Universal Monitor level.

This field is included as an Event Attribute.

Introduced in 1.0.0. Renamed to Event Tag in 2.0.0

Cloud Data Monitor Events

This monitor publishes Universal Events when objects are monitored successfully, based on the selected Action. The suggested implementation to accomplish this behavior is to associate a Cloud Data Monitor Task (event publisher) with a Universal Monitor Task (event consumer).

The Universal Event for this Universal Extension is described below.

Event Template: Event Monitor

Attribute

Description

Type

Attribute

Description

Type

Storage System

The Storage System where the monitored object is stored. This is the input field Storage System.

Text

Storage Filepath

The Storage System where the monitored object is stored. This is the input field Storage Filepath.

Text

Filename

The name of the monitored object.

Text

Filesize

The size of the monitored object.

Integer

File Modified Time

The modification time of the monitored object.

Text

Event Tag

Allows task authors to define a 'tag' attribute for the generated Universal Events, enabling more filtering capabilities at the Universal Monitor level. This is the Event Tag field.