Disclaimer

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at Stonebranch Integration Hub - Terms of Use.

Version Information

Refer to Changelog for version history information.

Overview

One of the key use cases for Service Orchestration and Automation Platforms is the ability to detect changes in files or directories and respond to these events by executing specific actions.

This integration enables monitoring of files and directories across various local and cloud storage systems. When a change is detected, Local Universal Events are triggered and can be processed within Universal Controller, such as initiating a task or workflow. 

Key Features

This Universal Extension supports the following key features:

Requirements

This integration requires a Universal Agent, and a Python runtime to execute the Universal Task. The following are the requirements for Version 2

Supported Actions

This integration supports the following actions.

Action: Monitor On Create

This action enables monitoring for the creation of one or multiple objects. The object can either be a file or a directory. Selecting this action enables the Trigger On Existence option, which treats the initial files of the directory as a file creation event. If this option is enabled, the user has to set a filter based on rclone filtering rules.

Configuration examples

Action Output

This extension provides EXTENSION output on Cancel or on Failure.

{
	"exit_code": 0,
	"status_description": "Task executed successfully",
	"invocation": {
		"extension": "ue-cloud-dm",
		"version": "2.0.0",
		"fields": {
			"action": "Monitor On Create",
			"trigger_on_existence": false,
			"configuration_file": "/home/user/data/tmp/6e9d8fd1-167c-4e74-b210-898d1eaaebef.con",
			"storage_system": "local_ua",
			"storage_filepath": "/home/ue-dev/USERS/user/cloud-dm/",
			"files_only": false,
			"storage_credentials": {
				"user": "599910b6-1234-1234-abcdef-1234cc63bd0",
				"password": "****",
				"token": "****",
				"key_location": null,
				"passphrase": "****"
			},
			"update_credentials": false,
			"controller_url": null,
			"controller_credentials": null,
			"refresh_storage_credentials": null,
			"monitor_time_interval": 10,
			"use_storage_modtime": false,
			"recursion_depth": 1,
			"use_filter": "Include",
			"filter": "{{report.*}}",
			"additional_options": null,
			"tag": null,
			"correlation_id": null,
			"log_level": "DEBUG",
			"configuration_file_name": "rclone.conf",
			"storage_credentials_user": null,
			"storage_credentials_password": null,
			"storage_credentials_token": null,
			"storage_credentials_name": "local_credentials",
			"controller_credentials_user": null,
			"controller_credentials_password": null,
			"controller_credentials_name": null
			}
		},
	"result": {
		"cmd": "/home/user/data/uag/extensions/.ue-cloud-dm/vendor/rclone lsjson 
               local_ua:/home/ue-dev/USERS/user/cloud-dm/ 
               --config=/home/user/data/tmp/2a100d7f-08b5-43c2-b20d-344a2c2d85a9.con 
               --max-depth=1 --log-level=DEBUG" --include {{report.*}}",
		"rc": 0
	}
}

{
	"exit_code": 1,
	"status_description": "Execution Failed: See STDERR for more information.",
	"invocation": {
		"extension": "ue-cloud-dm",
		"version": "2.0.0",
		"fields": {
			"update_credentials": false,
			"configuration_file": "/home/user/data/tmp/2a100d7f-08b5-43c2-b20d-344a2c2d85a9.con",
			"refresh_storage_credentials": null,
			"recursion_depth": 1,
			"tag": "extensions_test",
			"monitor_time_interval": 300,
			"use_filter": "-- None --",
			"additional_options": null,
			"files_only": false,
			"controller_url": null,
			"storage_system": "one_drive",
			"use_storage_modtime": false,
			"trigger_on_existence": true,
			"action": "Monitor On Create",
			"filter": null,
			"storage_filepath": "/user-storage/",
			"log_level": "DEBUG",
			"storage_credentials": {
				"user": "599910b6-1234-1234-abcdef-1234cc63bd0",
				"password": "****",
				"key_location": null,
				"passphrase": "****",
				"token": "****"
			}
		}
	},
	"result": {
		"cmd": "/home/user/data/uag/extensions/.ue-cloud-dm/vendor/rclone lsjson 
               one_drive:/user-storage/ 
               --config=/home/user/data/tmp/2a100d7f-08b5-43c2-20d-344a2c2d85a9.con
               --max-depth=1 --log-level=DEBUG",
		"rc": 1
		"errors": [
			"Failed to create file system for \"one_drive:/user-storage/\": 
			failed to get root: Get \"https://graph.microsoft.com/v1.0/drives/HASH/root\": 
			couldn't fetch token: invalid_client: if you're using your own client id/secret, 
			make sure they're properly set up following the docs"
		]
	}
}

Action: Monitor On Change and Monitor On Delete

The Monitor On Change action monitors if the object's modification time has been updated, while the Monitor On Delete action monitors whether an object has been deleted. These two actions share the same configuration.

Configuration examples

Action Output

This extension provides EXTENSION output on Cancel, or on Failure.

{
	"exit_code": 0,
	"status_description": "Task executed successfully",
	"invocation": {
		"extension": "ue-cloud-dm",
		"version": "2.0.0",
		"fields": {
			"action": "Monitor On Change",
			"trigger_on_existence": false,
			"configuration_file": "/home/user/data/tmp/6e9d8fd1-167c-4e74-b210-898d1eaaebef.con",
			"storage_system": "local_ua",
			"storage_filepath": "/home/ue-dev/USERS/user/cloud-dm/",
			"files_only": false,
			"storage_credentials": {
				"user": "599910b6-1234-1234-abcdef-1234cc63bd0",
				"password": "****",
				"token": "****",
				"key_location": null,
				"passphrase": "****"
			},
			"update_credentials": false,
			"controller_url": null,
			"controller_credentials": null,
			"refresh_storage_credentials": null,
			"monitor_time_interval": 10,
			"use_storage_modtime": false,
			"recursion_depth": 1,
			"use_filter": "Include",
			"filter": "{{report.*}}",
			"additional_options": null,
			"tag": null,
			"correlation_id": null,
			"log_level": "DEBUG",
			"configuration_file_name": "rclone.conf",
			"storage_credentials_user": null,
			"storage_credentials_password": null,
			"storage_credentials_token": null,
			"storage_credentials_name": "local_credentials",
			"controller_credentials_user": null,
			"controller_credentials_password": null,
			"controller_credentials_name": null
			}
		},
	"result": {
		"cmd": "/home/user/data/uag/extensions/.ue-cloud-dm/vendor/rclone lsjson
        one_drive:/user-storage/ 
        --config=/home/user/data/tmp/2a100d7f-08b5-43c2-20d-344a2c2d85a9.con
        --max-depth=1 --log-level=DEBUG",
		"rc": 0
	}
}

{
	"exit_code": 1,
	"status_description": ""Execution Failed: See STDERR for more information.",
	"invocation": {
		"extension": "ue-cloud-dm",
		"version": "2.0.0",
		"fields": {
		"update_credentials": true,
		"configuration_file": "/home/user/data/tmp/fa02639e-e7c4-4f26-90db-78a7b09acce8.con",
		"refresh_storage_credentials": null,
		"recursion_depth": 1,
		"tag": "extensions_test",
		"monitor_time_interval": 300,
		"use_filter": "-- None --",
		"additional_options": null,
		"files_only": false,
		"controller_url": "http://controller.stonebranch.org:8080/uc/",
		"storage_system": "one_drive",
		"use_storage_modtime": false,
		"trigger_on_existence": false,
		"action": "Monitor On Delete",
		"filter": null,
		"storage_filepath": "/invalid_directory/",
		"log_level": "DEBUG",
		"controller_credentials": {
			"user": "ops.****",
			"password": "****",
			"key_location": null,
			"passphrase": "****",
			"token": "****"
		},
		"storage_credentials": {
			"user": "12342de-1234-1234-abcdef-1234cc6fff",
			"password": "****",
			"key_location": null,
			"passphrase": "****",
			"token": "****"
			}
		}
	},
	"result": {
		"cmd": "/home/user/data/uag/extensions/.ue-cloud-dm/vendor/rclone lsjson 
               one_drive:/invalid_directory/
               --config=/home/user/data/tmp/fa02639e-e7c4-4f26-90db-78a7b09acce8.con
               --max-depth=1 --log-level=DEBUG",
		"rc": 3
 		"errors": [
			"error listing: directory not found",
			"Failed to lsjson with 2 errors: last error was: 
            error in ListJSON: directory not found"
		]
	}
}

Input Fields

The input fields for this Universal Extension are described below.

Cloud Data Monitor Events

This monitor publishes Universal Events when objects are monitored successfully, based on the selected Action. The suggested implementation to accomplish this behavior is to associate a Cloud Data Monitor Task (event publisher) with a Universal Monitor Task (event consumer).

The Universal Event for this Universal Extension is described below.

Event Template: Event Monitor

Task Examples

Two scenarios of the association between a Cloud Data Monitor Task (event publisher) with a Universal Monitor Task (event consumer) are described below. For both scenarios, a Cloud Data Monitor Task (Publisher) and a Universal Monitor Task (Consumer) should be configured properly.

Workflow Scenario
Trigger Scenario

Universal Monitor Task Example

Universal Monitor can be configured in different ways:

Workflow Example

Within a Workflow, only the Universal Monitor Task is configured and is a direct workflow child. The Cloud Data Monitor Task is directly associated with the Universal Monitor Task but is not in the scope of a Workflow.

Upon a successful monitor, the Monitor will go into Success status and the underlying Cloud Data Monitor will go into Finished status.

Trigger Example

This Use Case is when we want to monitor changes continuously and for each event to trigger a task or a workflow. 

Output Fields

Environment Variables

Environment Variables can be set in the Environment Variables task definition table.

SSL Verification

When attempting to Refresh Storage Credentials using HTTPS, it might be necessary to provide a CA Bundle. To do so, set the REQUESTS_CA_BUNDLE environment variable to the path of the desired CA Bundle.

Exit Codes

The exit codes for this Universal Extension are described below.

STDOUT and STDERR

Observability

Metrics are generated and exported by this integration, related to the generated Cloud Data Monitor Events.

Metric: ue.cdm.events

Metric Attributes

The following attributes are applicable for ue.cdm.events.

Other Observability Configuration Options

Administrators can update the Universal Event Template to control additional attributes sent out for Universal Event-based Metrics. However, this configuration should be approached with caution. When the possible distinct values of those attributes are high, that might lead to high cardinality issues.

Administrators can activate them with caution as follows:

1. Go to Administration → Universal Templates.

2. Select the Universal Template.

3. Click the “Event Templates” Tab.

4. Select the Name of the Event Template you want to update, which represents the metric you are interested in.

5. Update the “Optional Metric Labels” list of required Metric Labels.

Grafana Dashboard

A ready-to-use Grafana dashboard can be imported that enables enhanced visibility and understanding of the behavior for Task Instances.  It uses the above published metrics.

See the related How To section on how to install it.

How To

Import Universal Template

To use the Universal Template, you first should perform the following steps.

1. This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.

2. To import the Universal Template into your Controller, follow the instructions from the official documentation: Import An Integration.

3. When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list.

Modifications of this integration, applied by users or customers, before or after import, might affect the supportability of this integration. For more information refer to Integration Modifications.

Configure Universal Task

To configure a new Universal Task, there are three steps required:

• Create required Resolvable Credentials. Required as Input Fields on the Universal Task Configuration.
• Create a new Script of type Data, for the Configuration File and populate it according to the following section of Setup Rclone Configuration File as UAC Script.
• Create a new task, and enter the task-specific details that were created in the Universal Template.

Setup Rclone Configuration File as UAC Script

This integration depends on the Rclone binary which needs a configuration file to locate the required information for the involved storage system(s). This file is passed from the Universal Controller to the Universal Agent as UAC Script through the Configuration File Input Field. The Universal Controller Script should be of type Data, and option Resolve UAC Variables should be enabled.

Integrated Storage Systems & Sample Configurations

Rclone is integrated with multiple storage systems. This Universal Extension has been tested against the following Storage Systems.

• Amazon S3
• Google Cloud Storage
• Microsoft OneDrive Business
• Local file system (Linux, Windows)

The indicative Rclone configuration examples below contain the basic connection parameters.

[aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_storage_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_storage_credentials}')}
region = us-east-2

[google_cloud_storage]
type = google cloud storage
service_account_file = ${_credentialPwd('${ops_ue_cloud_storage_credentials}')}
object_acl = bucketOwnerFullControl
project_number = johnprojectno
location = europe-west3

[one_drive]
type = onedrive
token = ${_credentialToken('${ops_ue_cloud_storage_credentials}')}
client_id = ${_credentialUser('${ops_ue_cloud_storage_credentials}')}
client_secret = ${_credentialPwd('${ops_ue_cloud_storage_credentials}')}
drive_id = b!reXzf-NsJk-SAMPLE-DRIVE-ID-dfA_bM-DrjNN_hR5pD0u13Fvhi
tenant = a23424587-SAMPLE-TENANT-ID-cd34248adc
client_credentials = true
drive_type = business

[one_drive]
type = onedrive
token = ${_credentialToken('${ops_ue_cloud_storage_credentials}')}
client_id = ${_credentialUser('${ops_ue_cloud_storage_credentials}')}
client_secret = ${_credentialPwd('${ops_ue_cloud_storage_credentials}')}
drive_id = b!reXzf-NsJk-THISISASAMPLEDRIVEIDdfA_bM-DrjNN_hR5pD0u13Fvhi
drive_type = business

[linux]
type = local

[windows]
type = local

Using Credentials

Storage system configuration embodies the related credentials for authentication and authorization. It is advised that account credentials, tokens, or any other essential information inside the Rclone Configuration File be passed as Universal Controller Credentials, through Credential functions to be kept safe. 

There are several ways that Credential Functions can be used. To make the Configuration File immune to changes in credential names used in the task definition, it is recommended to follow the Configuration through Credential Field Variables approach. Another benefit of this approach is that it is compatible with all the Universal Controller versions supported by this integration. However, alternative approaches are Configuration through Credential Name and Configuration through Credential as Variable, which rely on changes performed on the Configuration File itself, rather than the task definition. The next chapters provide more information on them. 

For local filesystem storage, no credentials are required in the Configuration File. In this case, the credentials that are provided in the Agent Details of the Universal Task are used. If not, the selected action is executed with the permissions of the user that runs the ubroker daemon.

Configuration through Credential Field Variables

The table below lists the variables that can be used for the respective input Credential Field and an example of a function that can be used in the Configuration File definition.

Example of a Configuration File for AWS S3 Storage

[aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_storage_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_storage_credentials}')}
region = us-east-2

Configuration through Credential Name

In this case, the Credential Name is used directly inside the Configuration File.

Example of a Configuration File for AWS S3 Storage, using a Credential Function with the Credential Name

[aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('ue-aws-s3')}
secret_access_key = ${_credentialPwd('ue-aws-s3')}
region = us-east-2

When using this approach, always have in mind that:

1. Any update on the Storage Credential attributes will not affect the Configuration File. However, if different Credential Names are used on the Task definition, then the configuration file needs to be updated as well to refer to the new Credential names. 
2. A single change in the Configuration File will affect all the Cloud Data Monitor Tasks that are utilizing this Script. 

Configuration through Credential as Variable

Starting from Universal Controller 7.6, Credentials can be provided as UAC Variables:

The above approach requires no custom changes in the Configuration File contents.

[aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_storage_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_storage_credentials}')}
region = us-east-2

Refresh Storage Credentials

Refresh token for Microsoft OneDrive

Remote storages of type Microsoft OneDrive & Sharepoint use an OAuth2 token ({"access_token": ..., "token_type": ... , "refresh_token": ... , "expiry": .... }) to authorize users or applications. In this case, and only when the access token has expired, Rclone exchanges the existing refresh token for a new access token and then establishes a connection with the remote storage. This integration provides the capability to save the new value of the token on the selected Universal Controller Credential, so it can be used and configured as a Credential Function.

This integration provides an automatic update of the token when the identified Storage System is of type "onedrive" and it is enabled, and when the task input fields Update Credentials, Controller URL, Controller Credentials are selected.   

An example of such a configuration can be found below:

Refresh authorization fields for other remotes

Some remote storages might have the functionality of refreshing an authorization field. This integration provides the capability to save the new value of the authorization field on a selected Universal Controller Credential, so it can be used and configured as a Credential Function.

The task input fields related to this configuration are Update Credentials, Controller URL, Controller Credentials, and Refresh Storage Credentials.

An example of such a configuration can be found below:

Import Grafana Dashboard

Users can benefit from a ready-to-use sample dashboard that this downloadable integration offers when observability features are used. It is located under the /observability/grafana/ directory inside the downloadable zip file from the Stonebranch Integration Hub. Administrators should refer to the official Grafana User Guide on how to import a Grafana Dashboard.

Integration Modifications

Modifications applied by users or customers, before or after import, might affect the supportability of this integration. The following modifications are discouraged to retain the support level as applied for this integration.

• Python code modifications should not be done.
• Template Modifications

  • General Section

    • "Name", "Extension", "Variable Prefix", and "Icon" should not be changed.

  • Universal Template Details Section

    • "Template Type", "Agent Type", "Send Extension Variables", and "Always Cancel on Force Finish" should not be changed.

  • Result Processing Defaults Section

    • Success and Failure Exit codes should not be changed.
    • Success and Failure Output processing should not be changed.

  • Fields Restriction Section
    The setup of the template does not impose any restrictions, however, concerning the "Exit Code Processing Fields" section.

    1. Success/Failure exit codes need to be respected.
    2. In principle, as STDERR and STDOUT outputs can change in follow-up releases of this integration, they should not be considered reliable sources for determining the success or failure of a task.
    3. In Event Templates fields "Name", "Time To Live", "Unmapped Attribute Policy", "Attribute Name", and "Attribute Type", should not be changed.
       
       

    The suggested configuration for this Universal Extension does not require any updates on Time To Live. However, this field can be tuned for troubleshooting purposes, affecting immediately the time (in minutes) that events are stored in the Controller's Event Queue. Configure a low Time To Live value (for example 1 minute) if it is required to review the published Events.

    Use the Server Operation Inspect Universal Events to view all Universal Events in the Controller's event queue.

Users and customers are encouraged to report defects, or feature requests at Stonebranch Support Desk.

Known Limitations

• To properly monitor object renaming, the users have to use Action "Monitor On Create" instead of "Monitor on Change".

• In version 1, only the agent user (the user running the Universal Agent) could run this integration on Linux OS. In version 2.0.0, a workaround is available for Agents running without defect D-13034 resolved (updated prior to December 2024). To allow users other than the agent user to run a Cloud Data Monitor Task, follow these steps:

  1. Identify the Linux Agents on which the Cloud Data Monitor Task needs to run.

  2. Import the latest version of ue-cloud-dm (currently 2.0.0) on the desired Universal Controller.

  3. Initiate a test Cloud Data Monitor task on the agents identified in step (1), using the agent user (by leaving the Agent Details → Credentials field empty). This task ensures the integration installs correctly on each agent, enabling future tasks to run smoothly under various user accounts.

Migration Guide

Upgrading from version 1.0 to 2.0

When upgrading major versions, the following actions should be performed to migrate task definitions properly, due to the breaking changes:

• Rclone does not need to be installed on the agent system. It is bundled with the extension.
• The Correlation ID field has been renamed to Event Tag, no change is required.
• Tasks that refer to Cloud Data Monitor Task Instance Variables (for example sibling tasks within a Workflow), should be updated to ${ops_ue_cloud_<field_name>}. (the "_dm" part existing on version 1 has been removed)
• Configuration File should be updated when Credential Fields variables are referenced, as the variable names do not include "_dm" on version 2. 

[aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_dm_storage_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_dm_storage_credentials}')}
region = us-east-2

[aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_storage_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_storage_credentials}')}
region = us-east-2

• Event Template Changes - The Event Templates of Cloud Data Monitor version 1 have been merged into a single template to improve user experience. To migrate to version 2, only the Event Monitor event template is required:

With this change, Universal Events should be filtered by Action name using the new "event_type" attribute. To see what this looks like, refer to the Universal Monitor Task Example.

Document References

This document references the following documents.

Changelog

ue-cloud-dm-2.0.0 (2025-02-06)

Deprecations and Breaking Changes 

• Breaking Change: Replaced multiple event templates (1 per action) with one template having the "event_type" attribute. (#41506)
• Breaking Change: Changed extension prefix to ue_cloud, offering the ability to use the same configuration with Cloud Data Transfer. (#36364)

Enhancements

• Rclone is now bundled with the extension. (#36181)
• Automatically attempt to refresh OneDrive Credentials when the field Update Credentials is set. (#40914)
• The directory attribute was included in the Universal Event Template to indicate whether a monitored object is a directory. (#42435)
• Introduced Observability metrics enabling enhanced visibility and understanding of behavior for Task Instances. (#42806)
  
• The option to skip SSL Certificate Verification when selecting the Update Credentials field. (#43388)
• Update Credentials on UC will be performed only when there is new information to Update. (#43708)
• Changed the label of the "Correlation ID" field to "Event Tag". (#41738)

Fixes

• Monitor On Change Index Error on file creation. (#41944)
• Issue with setting Recursion Depth value greater than 1 when files in different directories have the same name. (#42434)
• Extension Failure when the Use Storage Modtime field is not checked. (#42670)
• Permission issue on Linux OS which occurs when a task is executed with a user different than the agent user. (#42955)
• Key Location field is not retained when updating Universal Controller Credentials. (#43411)
• Task failure when using Windows paths with whitespace. (#43610)

ue-cloud-dm-1.0.1 (2024-08-14)

Fixes

• Events were not published with the Action "Monitor On Create" when Trigger on Existence was enabled. (#41688)

• Warnings appeared at the beginning of the extension execution on later versions of the Universal Controller. (#36812)

• Issue when Updating Credentials. (#36117)

ue-cloud-dm-1.0.0 (2022-10-26)

Enhancements

• Initial version (#29658)