Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Panel

Table of Contents

Disclaimer

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at https://www.stonebranch.com/integration-hub/Terms-and-Privacy/Terms-of-Use/

Overview

This Universal Extension is an interface to Rclone that provides the capability to monitor files/directories from across different cloud storages, as well as local or distributed file systems. Additionally and upon successful monitor, this extension publishes Local Universal Events. To properly handle these events and take action on them, this Universal Extension optionally can be attached as a Publisher to a Universal Monitor Task (see more details in section Cloud Data Monitor Events).
Rclone, is the open source command line program, that is utilized to accomplish all the actions supported in this Extension.

Version Information

Template NameExtension NameExtension Version
Cloud Data Monitorue-cloud-dm1.0.1

Refer to Cloud Data Monitor Changelog for version history information.

Software Requirements

This integration requires a Universal Agent, a Python runtime to execute the Universal Task, and a Rclone executable.

Software Requirements for Universal Template and Universal Task

Requires Python 3.7.0 or higher. Tested with the Universal Agent bundled Python distribution.

Software Requirements for Universal Agent

Both Windows and Linux agents are supported.

  • Universal Agent for Windows x64 Version 7.2.0.0 and later with python options installed.
  • Universal Agent for Linux Version 7.2.0.0 and later with python options installed.

Software Requirements for Universal Controller

Universal Controller Version 7.2.0.0 and later.

Software Requirements for Rclone

Rclone needs to be installed on the same server where the Universal Agent is installed. Rclone binary should be stored on a location where Universal Agent has access and permissions to execute it.

Note

This Universal Extension has been tested with Rclone v1.58.1. It should be working with later Rclone versions, as long as these are backwards compatible.


Network and Connectivity Requirements

Universal Agent should be able to establish connection with:

  • The remote storages that are provided as input for the Universal Task.
  • The Universal Controller host. This is required only in case Refresh Storage Credentials field is used.

Key Features

This Universal Extension supports the following key features:

  • Actions
    • Monitor objects creation
    • Monitor objects change - change based on modification time
    • Monitor objects deletion
  • Options
    • Trigger On Existence - available for Action Monitor On Create.
    • Advanced filtering capability for objects to be monitored.
    • Support for providing additional Rclone options according to the user needs.
  • Universal Events
    • Event published upon single object creation, holding information about the new object.
    • Event published upon single object change, holding information about the updated file including the latest object size and modification time.
    • Event published upon single object deletion, holding information about the deleted object.
    • Event published upon single object existence, holding information about the object.

Integrated Storage Systems

Rclone is integrated with multiple storage systems. This Universal Extension has been tested against the following Storage Systems.

...

Note

This integration should work properly against other Storage Systems, as long as the integration interface (see chapter Configure Universal Task) and functionalities listed in this document fulfil the needs of the Storage System.

For functionalities required for specific Storage Systems, users and customers are encouraged to open a Feature Request in our Customer Support Portal.


Import Universal Template

To use the Universal Template, you first must perform the following steps.

...

Modifications of this integration, applied by users or customers, before or after import, might affect the supportability of this integration. For more information refer to Integration Modifications.

Configure Universal Task

To configure a new Universal Task, there are three steps required:

  • Create required Resolvable Credentials. Required as Input Fields on the Universal Task Configuration.
  • Create a new Script of type Data, for the Configuration File and populate it according to the following section of Setup Rclone Configuration File. Required as Input Fields on the Universal Task Configuration.
  • Create a new task, and enter the task-specific details that were created in the Universal Template.

Setup Rclone Configuration File

  1. The configuration file is following the INI format, and contains all required parameters and credentials to connect to a Storage System. All configuration options per Storage System can be found in the respective "Config Rclone documentation" .

  2. It is advised that account credentials, tokens, or any other essential information to be passed as an encrypted Resolvable Credential field. To set a value with the respective Credential field, please make sure to use the appropriate Credential function inside the configuration file.

    Note

    'Runtime Password' Credentials field supports values with up to 512 characters.

    Hard coded values can be provided in the script file, as well. However, it is not advised for security reasons.

  3. It is recommended to configure a storage system section, one to one with the Credential fields in this Universal Extension (please refer to the Input Fields section below).

  4. For local file system Storages, no credentials are required in the configuration file. The selected action will be executed with the Credentials that are provided in the "Agent Details" of the Universal Task. If no Credentials are provided for the used Agent, then the same user that runs ubroker daemon will be used.

  5. Script Type should be Data, and option Resolve UAC Variables should be enabled.

...

[ue_aws_s3]
type = s3
provider = AWS
access_key_id = ${_credentialUser('${ops_ue_cloud_dm_storage_credentials}')}
secret_access_key = ${_credentialPwd('${ops_ue_cloud_dm_storage_credentials}')}
region = us-east-2


[google_cloud_storage]
type = google cloud storage
service_account_file = ${_credentialPwd('${ops_ue_cloud_dm_storage_credentials}')}
object_acl = bucketOwnerFullControl
project_number = johnprojectno
location = europe-west3

[one_drive]
type = onedrive
token = ${_credentialToken('${ops_ue_cloud_dm_storage_credentials}')}
drive_id = ${_credentialUser('${ops_ue_cloud_dm_storage_credentials}')}
drive_type = business
auth_url = https://login.microsoftonline.com/<tenant id>/oauth2/v2.0/authorize
token_url = https://login.microsoftonline.com/<tenant id>/oauth2/v2.0/token

[local_filesystem]
type = local

Input Fields

The input fields for this Universal Extension are described below.

FieldInput typeDefault valueTypeDescription
ActionRequiredMonitor On CreateChoiceAction to be performed.

Available actions:
  • Monitor On Create
  • Monitor On Change
  • Monitor On Delete
Trigger On ExistenceOptionalFalseBooleanWhen this flag is enabled and objects already exist on the given Storage Filepath, an event will be published. Otherwise, Universal Task will keep monitoring for new objects creation.

Optional when Action Monitor On Create is selected.
Configuration FileRequired-ScriptRclone Configuration file, that contains all required parameters to connect to the Storage System.
Storage SystemOptional-Dynamic Choice FieldRemote or Local storage to execute the selected Action.
Execute the Dynamic Choice Field and retrieve all the available Storages from Configuration File.

This field is included as an Event Attribute.
Storage CredentialsOptional-CredentialCredentials needed to connect to the selected Storage System.

For Storage System of type local, the standard Agent's Credentials field should be provided.
Storage FilepathOptional-TextPath to monitor objects for the selected Action.

This field is included as an Event Attribute.
Update CredentialsOptionalFalseBooleanThis field needs to be used when Rclone executable updates Fields on its Configuration File during execution, that need to be stored as Credential Fields on the Universal Controller.

For example: when OneDrive Storage type is used. In this case, the existing configured token is exchanged by Rclone with a new one to establish the connection. This new token should be stored on Universal Controller for the next task execution. Please read Rclone limitations on OneDrive token refresh here.

This field is used in conjunction with fields Controller URL, Controller Credentials, Refresh Storage Credentials.
Controller URLOptional-TextController URL where Storage Credentials are stored.

Required when Update Credentials is checked.
Controller CredentialsOptional-CredentialsController user's credentials, used for logging and updating Storage Credentials will be updated. The Credentials definition should be as follows:
  • User as "Runtime User".
  • User Password as "Runtime Password".
Note: The provided user should have sufficient Credential permissions.

Required when Update Credentials is checked.
Refresh Storage CredentialsOptional-ArrayThis Array field is available when Update Credentials is set to True and used as a mapping table that indicates which field from the Configuration File of Rclone is used to update which Credential entry field on Universal Controller.

Populate this array with the following format:
  • Configuration File Details: "<Storage Storage>"."<Configuration File's field to be refreshed by Rclone>"
  • Storage Credentials Details: "<Storage Credentials name>"."<Storage Credentials field to be updated>"
The result of this action is that the Configuration file's field value, is copied to the provided Storage Credential's field. Each line of this Array is evaluated, and related mapping is performed.
Note: Refresh of Storage Credentials is performed via Controller's REST API. See Network and Connectivity Requirements
Monitor Time IntervalOptional300IntegerInterval period in seconds used to monitor the given Storage Filepath.
Use FilterOptional-- None --ChoiceThe filter type that is applied on the Action.

Available options:
  • -- None --
  • Include
  • Exclude
FilterOptional-TextFilter based on Rclone filtering rules that is applied on the Action.

Required when Use Filter is checked.
Files OnlyOptionalFalseBooleanWhen enabled, directories will be ignored and only files will be monitored.
Use Storage System Modified TimeOptionalTrueBooleanWhen enabled, the time of the Storage System will be observed as the object's modification time, instead of the original object's modified time stored as metadata.

Note: Having this option enabled, increases performance.
Recursion DepthOptional1IntegerRecursion depth that is applied. Default value 1 means that no recursion will be applied.
Additional OptionsOptional-TextSpace separated Rclone options that applied to the selected Action.
Correlation IDOptional-TextUse this field optionally to pass extra custom information to the published event (for example: provide the parent Workflow task instance ID).
This field is included as an Event Attribute.

Cloud Data Monitor Events

This Universal Extension publishes Universal Events when objects are monitored succesfully, based on the selected Action. The suggested implementation to accomplish this behavior is to associate a Cloud Data Monitor Task(event publisher) with a Universal Monitor Task(event consumer).

...

  • Event Monitor On Create

    AttributeDescriptionType
    Storage SystemThe Storage System where the monitored object is stored. This is the input field Storage System.Text
    Storage FilepathThe Storage System where the monitored object is stored. This is the input field Storage Filepath.Text
    FilenameThe name of the monitored object.Text
    Correlation IDCustom information field. This is the input field Correlation ID.Text


  • Event Monitor On Change

    AttributeDescriptionType
    Storage SystemThe Storage System where the monitored object is stored. This is the input field Storage System.Text
    Storage FilepathThe Storage System where the monitored object is stored. This is the input field Storage Filepath.Text
    FilenameThe name of the monitored object.Text
    FilesizeThe updated size of the monitored object.Text
    File Modified TimeThe time when the object last changed.Text
    Correlation IDCustom information field. This is the input field Correlation ID.Text


  • Event Monitor On Delete

    AttributeDescriptionType
    Storage SystemThe Storage System where the monitored object is stored. This is the input field Storage System.Text
    Storage FilepathThe Storage System where the monitored object is stored. This is the input field Storage Filepath.Text
    FilenameThe name of the monitored object.Text
    Correlation IDCustom information field. This is the input field Correlation ID.Text


  • Event Monitor On Create Exists

    AttributeDescriptionType
    Storage SystemThe Storage System where the monitored object is stored. This is the input field Storage System.Text
    Storage FilepathThe Storage System where the monitored object is stored. This is the input field Storage Filepath.Text
    FilenameThe name of the monitored object.Text
    File Modified TimeThe time when the object last changed.Text
    Correlation IDCustom information field. This is the input field Correlation ID.Text


Task Examples

Two scenarios of the appropriate association between Cloud Data Monitor Task(event publisher) with a Universal Monitor Task(event consumer), are described below.

Workflow Scenario

Trigger Scenario

For both scenarios, a Cloud Data Monitor Task (Publisher) and a Universal Monitor Task (Consumer) should be configured properly.

Below are some example configurations of those respective tasks.

Cloud Data Monitor Task Example

Universal Monitor Task Example

Universal Monitor Tasks can be associated to:

...

Note

In a Universal Monitor Task, Event Type should always be "Local". Monitor Criteria of a Universal Monitor Task, are the Attributes of the associated Event Template.


Workflow Example

Within a Workflow, only the Universal Monitor Task is configured, and is a direct workflow child. Cloud Data Monitor Task is directly associated with Universal Monitor Task, but is not in the scope of Workflow.

Upon a successful monitor, the Monitor will go into Success status and the underlying Cloud Data Monitor will go into Finished status.

Trigger Example

Task Output

Output Only Fields

The output fields for this Universal Extension are described below.

FieldTypeDescription
Total Event CountIntegerTotal number of the published Events.
Last Event TimestampTextTimestamp in UTC when the latest event got published.

Exit Codes

The exit codes for this Universal Extension are described below.

Exit CodeStatus Classification CodeStatus Classification DescriptionStatus Description
0SUCCESSSuccessful ExecutionSUCCESS: Cloud Data Monitor Task completed successfully.
1FAILFailed Execution
  • FAIL: Unexpected error. Execute in Debug for more information.
  • FAIL: Cloud Data Monitor task exited with non-zero exit code. See STDERR for more details.
20FAILData ValidationDATA VALIDATION ERROR: <informative_message>
22FAILFailed ExecutionFAIL: Cloud File Monitor task executed successfully but updating Credentials failed.

Extension Output

Note

This Universal Extension produces EXTENSION Output only when the Rclone command fails to be executed.

...

{
  "exit_code": 1,
  "status_description": "FAIL: Cloud Data Monitor task exited with non zero exit code. See STDERR for more details. ",
  "changed": false,
  "invocation": {
    "extension": "ue-cloud-dm",
    "version": "1.0.0",
    "fields": {
      "action": "Monitor On Create",
      "trigger_on_existence": false,
      "configuration_file_name": "monitor_rclone.conf",
      "log_level": "DEBUG",
      "storage_system": "ue_aws_s3_source",
      "storage_filepath": "john-bucket",
      "storage_credentials_name": "ue_aws_s3",
      "update_credentials": false,
      "controller_url": null,
      "controller_credentials_name": null,
      "refresh_storage_credentials": [],
      "files_only": false,
      "use_filter": null,
      "filter": null,
      "monitor_time_interval": 5,
      "use_storage_modtime": true,
      "recursion_depth": "1",
      "additional_options": [],
      "correlation_id": null
    }
  },
  "result": {
    "cmd": "rclone --config=/var/opt/universal/tmp/d5e99337-6c94-4c5a-bbd4-93fc43f11c03.con --max-depth=1 --log-level=DEBUG --use-server-modtime lsjson ue_aws_s3_source:john-bucket",
    "rc": 3
  }
}

STDOUT and STDERR

STDOUT and STDERR provide additional information to User. The populated content can be changed in future versions of this extension without notice. Backward compatibility is not guaranteed.

...

Anchor
Integration Modifications
Integration Modifications

Integration Modifications

Modifications applied by users or customers, before or after import, might affect the supportability of this integration. The following modifications are discouraged to retain the support level as applied for this integration.

...

Users and customers are encouraged to report defects, or feature requests at Stonebranch Support Desk.

Known Limitations

Monitor object renaming

To properly monitor on object renaming, use Action Monitor On Create rather Monitor on Change.

Document References

This document references the following documents.

Document LinkDescription
Universal TemplatesUser documentation for creating Universal Templates in the Universal Controller user interface.
Universal TasksUser documentation for creating Universal Tasks in the Universal Controller user interface.
Universal Event TemplatesUser documentation for Universal Event Template fields definition.
Universal Monitor TasksUser documentation for creating Universal Monitor Tasks in the Universal Controller user interface.
RcloneRclone official documentation.
Rclone InstallInstallation instructions.
Rclone DownloadsDownload links for Linux, Windows, MacOS.
Rclone Storage Systems configurationConfiguration file details.
Rclone Exit CodesRclone exit codes list.

Anchor
Changelog
Changelog

Changelog

ue-cloud-dm-1.0.1 (2024-08-14)

Fixed: Events were not published with the action Monitor On Create when Trigger on Existence was enabled. (#41688)

...

Fixed: Issue when Updating Credentials. (#36117)

ue-cloud-dm-1.0.0 (2022-10-26)

Added: Initial version (#29658)