Disclaimer

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at https://www.stonebranch.com/integration-hub/Terms-and-Privacy/Terms-of-Use/

Introduction

Storing data in the cloud becomes an integral part of most modern IT landscapes. With Universal Automation Center you can securely automate your AWS, Azure, Google and MinIO File Transfers and integrate them into your existing scheduling flows.

As security is one of the key concerns, when moving to the cloud, the provided solution supports multi-level of security:

Credentials for Azure Keys are stored in an encrypted form in the database
Support for Azure Token based Shared Access Signatures ( SAS )
Communication to Azure is done via the HTTPS protocol
A Proxy Server connection to Azure with basic authentication is supported

This Universal Task focuses on file transfer to, from and between an Azure Blob Storage.

A similar solution as for Azure Blob Storage is also available for AWS S3 and Google Cloud Storage.

Overview

The Universal Task for Azure Blob: Managed File Transfers allows to securely transfers files from, to and between Azure Blob Storage container and folders.

The Universal Task for Azure Blob: Managed File Transfers supports the following main features:

The following file transfer commands are supported:
- Upload a file(s) to an Azure Blob Storage container
- Download of file(s) from an Azure Blob Storage container
- Transfer files between Azure Blob Storage containers
- List objects in an Azure Blob Storage container
- Delete object(s) in an Azure Blob Storage container
- List Azure Blob Storage container names
- Create an Azure Blob Storage container
- Monitor a Blob in a Storage container
File Transfer can be triggered by a third-party application using the Universal Automation Center RESTfull Webservice API: REST API
Universal Task for Azure Blob Storage can be integrated into any existing scheduling workflow in the same way as any standard Linux or Windows Task type.
Security is ensured by using the HTTPS protocol with support for an optional Proxy Server.
Support for Azure Token based Shared Access Signatures ( SAS )
No Universal Agent needs to be installed on the Azure Cloud – the communication goes via HTTPS

Software Requirements

Software Requirements for Universal Agent

Universal Agent for Linux or Windows Version 7.0.0.0 or later are required
The Universal Agent needs to be installed with python option (--python yes)

Software Requirements for Universal Controller

Universal Controller 7.0.0.0. or later is required

Software Requirements for the Application to be Scheduled

The Universal Task has been tested for the python Azure Storage SDK: azure-storage-blob version: 12.7.1

Key Features

The solution supports the following file transfer scenarios:

Feature	Description
Upload a file(s) to a container (Copy or Move)	One or more files can be uploaded to a container. Move or copy are supported for the upload Unix filename pattern matching support; for example, wildcard support “” to upload multiple files One or more files can be Uploaded to a folder using a prefix It can be decided via an the Upload Write Options* to: Overwrite an existing object (Replace existing Object) Cancel the operations in case an object with a similar name exists (Do not overwrite existing Object) Add a timestamp to the uploaded Object (Timestamp)
Download of file(s) from a container (Copy or Move)	One or more files should be downloaded from a bucket. Move or copy must be supported for the download Unix filename pattern matching support; for example, wildcard support “” to download multiple files Download to a specific folder is supported It can be decided via an the Download Write Options* to: Overwrite an existing file (Replace existing File) Cancel the operations in case a File with a similar name exists (Do not overwrite existing File) Add a timestamp to the uploaded file (Timestamp) Perform the default Windows behaviour for copying files (Default Windows behaviour) If a file with a similar name exists, the file names that are similar will be edited so that the files you copied have a number appended at the end of them. For example, if you’re copying a file named image.png to a folder that already has a file named image.png in it, the copied file will be named image (1).png.
Copy object to a container	An object can be copied from one container to another. Folders are support Note: wildcards are not supported for this method.
List objects in a container	Show all Objects in a container Show all Objects in a folder Unix filename pattern matching support e.g. wildcard support “*” to narrow down the objects to display
Delete object(s) in a container	Delete one or multiple objects in a container Delete one or multiple objects in folder in a container Unix filename pattern matching support e.g. wildcard support “*” to narrow down the objects to be deleted
List buckets	List all container in an Azure account
Create a container	Create a new container
Monitor Blob	Monitor a Blob in a Storage container
Proxy Server Connection	A proxy server connection with or without basic authentication can be configured
Integration into 3^rd Party Applications	An Azure Blob Storage file transfer can be triggered via the Universal Automation Center RESTfull Webservice API within an application.
Self-Service through Web-Client	The Azure Blob Storage Task can be fully configured, monitored and updated via the Universal Controller Web-GUI

Import Azure Blob Storage Universal Template

The Universal Template is provided as a Zip-file. This Zip-file can be imported from any local folder using the Universal Controller UI.

To load the Universal Template Zip-file perform the following steps:

This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.
Copy the Universal Template Zip-file to a local directory.
In the Universal Controller UI, select Administration > Universal Templates > Import Template.
Select the Universal Template Zip-file to import.

When the Universal Template Zip-file has been imported successfully, the Universal Template will appear on the list.

Configure Azure Blob Storage Universal Tasks

For the new Universal Task type, create a new task and enter the task-specific Details that were created in the Universal Template.

Field Descriptions for Azure Blob Storage Universal Task - Action

The Azure Blob Storage Task provides multiple different file transfer actions. For each action the specific fields are described.

Azure Blob Storage Container is abbreviated to Container in the following.

Create Container - Action

The Action Creates a new Container.

Field	Description
Action	create-container action
Useproxy ( default is NO )	[NO \| YES] If set to YES, the fields to set-up the proxy server connections are displayed: Proxy Server IP or hostname Proxy Server Port Proxy Server Credentials (optional)
Loglevel ( default is INFO )	Universal Task logging settings [DEBUG \| INFO\| WARNING \| ERROR \| CRITICAL]
Connection Type	[Azure Account Key \| SAS Token ] If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration. If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.
Container	Name of the Container to create

Example for Azure Blob Storage Universal Tasks - Create Container

The following Task creates the Container: stonebranchpm2

List Container - Action

The Action list all container of an Azure Account.

Field	Description
Action	list-container action
Connection Type	[Azure Account Key \| SAS Token ] If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration. If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.
Useproxy ( default is NO )	[NO \| YES] If set to YES, the fields to set-up the proxy server connections are displayed: Proxy Server IP or hostname Proxy Server Port Proxy Server Credentials (optional)
Loglevel ( default is INFO )	Universal Task logging settings [DEBUG \| INFO\| WARNING \| ERROR \| CRITICAL]

Example for Azure Blob Storage Universal Tasks - List Containers

The following Task list all container of the selected Azure Account.

Upload File - Action

The Action is used to upload a single or multiple files from a Windows or Linux server to an container or a folder in a container.

Field	Description
Action	Upload-file action
Connection Type	[Azure Account Key \| SAS Token ] If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration. If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.
Useproxy ( default is NO )	[NO \| YES] If set to YES, the fields to set-up the proxy server connections are displayed: Proxy Server IP or hostname Proxy Server Port Proxy Server Credentials (optional)
Loglevel ( default is INFO )	Universal Task logging settings [DEBUG \| INFO\| WARNING \| ERROR \| CRITICAL]
Sourcefile	Source file(s) to be uploaded to a bucket or specific folder in a container. Unix filename pattern are supported to upload a selection of files. Supported wildcards are: ? matches any single character [seq] matches any character in seq [!seq] matches any character not in seq ”*” matches everything
Operation	[copy \| move] In case of a “move” the source file(s) is/are deleted after the upload to the container.
Prefix	A folder in a container is called prefix. In the field Prefix only the name of the folder needs to be provided no “/” after the folder name.
Upload Write Options	Upload Write Options [Do not overwrite existing Object \| Timestamp] Do not overwrite existing Object: Cancel the operations in case an object with a similar name exists Timestamp: Add a timestamp to the uploaded Object

Example for Azure Blob Storage Universal Tasks - Upload File

The following Task uploads from the Linux directory the files: /home/stonebranch/demo/azure/out/test[1-4]* to the the container stonebranchpm, folder incoming.

List Objects - Action

The Action is used to display objects in a Azure Container or a specific folder in an Azure Container (prefix).

Field	Description
Action	List-objects action
Connection Type	[Azure Account Key \| SAS Token ] If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration. If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.
Useproxy ( default is NO )	[NO \| YES] If set to YES, the fields to set-up the proxy server connections are displayed: Proxy Server IP or hostname Proxy Server Port Proxy Server Credentials (optional)
Loglevel ( default is INFO )	Universal Task logging settings [DEBUG \| INFO\| WARNING \| ERROR \| CRITICAL]
Container	Container in which the objects should be listed
Prefix	A folder in a container is called prefix. In the field Prefix only the name of the folder needs to be provided no “/” after the folder name. If a prefix is provided only objects in the folder with the prefix name are listed in the output.
Blob	Objects matching the given Blob are listed. Unix filename pattern are supported to list only a selection of files: Supported wildcards are: ? matches any single character [seq] matches any character in seq [!seq] matches any character not in seq ”” matches everything Example: Blob = test : matches everything starting with test Blob = test[1-2].txt : matches test1.txt, test2.txt Blob = test[!1].txt: does not match test1.txt Blob = test?.txt: matches test1.txt, test2.txt etc.
Show Details	Show details like creation timestamp in the output

Example for Azure Blob Storage Universal Tasks - List Objects

The following Task list all objects matching the criteria test* in the container stonebranchpm in the folder incoming.

Download File - Action

This Action downloads one or multiple files from an Azure Container to a Linux or Windows folder.

Field	Description
Action	Download-file action
Connection Type	[Azure Account Key \| SAS Token ] If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration. If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.
Useproxy ( default is NO )	[NO \| YES] If set to YES, the fields to set-up the proxy server connections are displayed: Proxy Server IP or hostname Proxy Server Port Proxy Server Credentials (optional)
Loglevel ( default is INFO )	Universal Task logging settings [DEBUG \| INFO\| WARNING \| ERROR \| CRITICAL]
Target Directory	Linux or Windows Target Directory For example, C:\tmp\ or /home/ubuntu/download Unix filename pattern are supported to download a selection of files. Supported wildcards are: ? matches any single character [seq] matches any character in seq [!seq] matches any character not in seq ”*” matches everything
Operation	[copy \| move] In case of a “move” the objects are deleted after they have been download from the container.
Download Write Options	Download Write Options: [Replace existing File \| Do not overwrite existing File \| Timestamp \| Default Windows behaviour] Replace existing File : overwrite an existing file Do not overwrite existing File: cancel the operations in case a File with a similar name exists Timestamp: add a timestamp to the uploaded file Default Windows behaviour: perform the default Windows behaviour for copying files. If a file with a similar name exists, the file names that are similar will be edited so that the files you copied have a number appended at the end of them. For example, if you are copying a file named image.png to a folder that already has a file named image.png in it, the copied file will be named image (1).png.

Example for Azure Blob Storage Universal Tasks - Download File

The following Task downloads from the container stonebranchpm all blobs matching the criteria test[1-2]* to the Linux directory: /home/stonebranch/demo/azure/in

Delete Objects - Action

The Actions is used to delete an object in an Azure Container or folder

Field	Description
Action	Delete-objects action
Connection Type	[Azure Account Key \| SAS Token ] If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration. If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.
Useproxy ( default is NO )	[NO \| YES] If set to YES, the fields to set-up the proxy server connections are displayed: Proxy Server IP or hostname Proxy Server Port Proxy Server Credentials (optional)
Loglevel ( default is INFO )	Universal Task logging settings [DEBUG \| INFO\| WARNING \| ERROR \| CRITICAL]
Container	Container in which the Blobs should be deleted.
Blob	Blobs to be deleted in the given container Note: Due to security reasons, wildcard is only supported if at least one character is provided e.g. t* would delete all files starting with at “t”.

Example for Azure Blob Storage Universal Task - Delete Objects

The following Task deletes in the container stonbranchpm, folder incoming all Blobs matching the criteria test[3-4]*.

Copy Object to Container - Action

This Action is used copy an objects from one Azure Container to another Azure Container.

Field	Description
Action	Copy-object-to-bucket action
Connection Type	[Azure Account Key \| SAS Token ] If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration. If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.
Useproxy ( default is NO )	[NO \| YES] If set to YES, the fields to set-up the proxy server connections are displayed: Proxy Server IP or hostname Proxy Server Port Proxy Server Credentials (optional)
Loglevel ( default is INFO )	Universal Task logging settings [DEBUG \| INFO\| WARNING \| ERROR \| CRITICAL]
Container	Source container
Target Container	Target container, where the object(s) will be copied to
Blob	File ( Blob ) to be copied from the source container to target container.
Upload Write Options	Upload Write Options [Do not overwrite existing Object \| Timestamp] Do not overwrite existing Object: Cancel the operations in case an object with a similar name exists Timestamp: Add a timestamp to the uploaded Object

Example for Azure Blob Storage Universal Tasks - Copy Object to Container

The following Task copies the Blob: test1.txt from the source container: stonebranchpm to the target container: stonebranchpm2

Delete Container - Action

This action is used to delete an Azure Container

Field	Description
Action	Delete-container action
Connection Type	[Azure Account Key \| SAS Token ] If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration. If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.
Useproxy ( default is NO )	[NO \| YES] If set to YES, the fields to set-up the proxy server connections are displayed: Proxy Server IP or hostname Proxy Server Port Proxy Server Credentials (optional)
Loglevel ( default is INFO )	Universal Task logging settings [DEBUG \| INFO\| WARNING \| ERROR \| CRITICAL]
Container	Name of the Container to be deleted Note: The container is delete including all objects.

Example for Azure Blob Storage Universal Tasks - Delete Container

The following Task delete the container stonebranchpm2

Upload File - Action

The Action is used to upload a single or multiple files from a Windows or Linux server to an container or a folder in a container.

Field	Description
Action	Upload-file action
Connection Type	[Azure Account Key \| SAS Token ] If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration. If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.
Useproxy ( default is NO )	[NO \| YES] If set to YES, the fields to set-up the proxy server connections are displayed: Proxy Server IP or hostname Proxy Server Port Proxy Server Credentials (optional)
Loglevel ( default is INFO )	Universal Task logging settings [DEBUG \| INFO\| WARNING \| ERROR \| CRITICAL]
Sourcefile	Source file(s) to be uploaded to a bucket or specific folder in a container. Unix filename pattern are supported to upload a selection of files. Supported wildcards are: ? matches any single character [seq] matches any character in seq [!seq] matches any character not in seq ”*” matches everything
Operation	[copy \| move] In case of a “move” the source file(s) is/are deleted after the upload to the container.
Prefix	A folder in a container is called prefix. In the field Prefix only the name of the folder needs to be provided no “/” after the folder name.
Upload Write Options	Upload Write Options [Do not overwrite existing Object \| Timestamp] Do not overwrite existing Object: Cancel the operations in case an object with a similar name exists Timestamp: Add a timestamp to the uploaded Object

Example for Azure Blob Storage Universal Tasks - Upload File

The following Task uploads from the Linux directory the files: /home/stonebranch/demo/azure/out/test[1-4]* to the the container stonebranchpm, folder incoming.

Monitor Blob - Action

The Action is used to monitor a Blob in an Azure Container.

Field	Description
Action	Monitor-Blob action
Connection Type	[Azure Account Key \| SAS Token ] If set to ‘Azure Account Key’, the credential field ‘Azure Account’ will appear for configuration. If set to ‘SAS Token’, the credential field ‘SAS Token’ will appear for configuration.
Useproxy ( default is NO )	[NO \| YES] If set to YES, the fields to set-up the proxy server connections are displayed: Proxy Server IP or hostname Proxy Server Port Proxy Server Credentials (optional)
Loglevel ( default is INFO )	Universal Task logging settings [DEBUG \| INFO\| WARNING \| ERROR \| CRITICAL]
Container	Container in which the objects should be monitored
Prefix	A folder in a container is called prefix. In the field Prefix only the name of the folder needs to be provided no “/” after the folder name. If a prefix is provided only objects in the folder with the prefix name are listed in the output.
Blob	Blob to monitor. The task goes to status success, if the Blob is found. Unix filename pattern are supported to list only a selection of files: Supported wildcards are: ? matches any single character [seq] matches any character in seq [!seq] matches any character not in seq ”” matches everything Example: Blob = test : matches everything starting with test Blob = test[1-2].txt : matches test1.txt, test2.txt Blob = test[!1].txt: does not match test1.txt Blob = test?.txt: matches test1.txt, test2.txt etc.

Example for Azure Blob Storage Universal Tasks - Monitor Blob

The following Task monitors in the Container: stonebranchpm all Blob’s starting with the name report. If a Blob starting with the name report is found the task goes to task status success.