/
Azure Blob: Manage File Transfers

Azure Blob: Manage File Transfers

CUSTOMER GENERIC

Disclaimer

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at Stonebranch Integration Hub - Terms of Use

Introduction

Storing data in the cloud becomes an integral part of most modern IT landscapes. With Universal Automation Center you can securely automate your AWS, Azure, Google and MinIO File Transfers and integrate them into your existing scheduling flows.

As security is one of the key concerns, when moving to the cloud, the provided solution supports multi-level of security:

  • Credentials for Azure Keys are stored in an encrypted form in the database

  • Support for Azure Token based Shared Access Signatures ( SAS )

  • Communication to Azure is done via the HTTPS protocol

  • A Proxy Server connection to Azure with basic authentication is supported

This Universal Task focuses on file transfer to, from and between an Azure Blob Storage.

A similar solution as for Azure Blob Storage is also available for AWS S3 and Google Cloud Storage.

Overview

The Universal Task for Azure Blob: Managed File Transfers allows to securely transfers files from, to and between Azure Blob Storage container and folders.

The Universal Task for Azure Blob: Managed File Transfers supports the following main features:

  • The following file transfer commands are supported:

    • Upload a file(s) to an Azure Blob Storage container

    • Download of file(s) from an Azure Blob Storage container

    • Transfer files between Azure Blob Storage containers

    • List objects in an Azure Blob Storage container

    • Delete object(s) in an Azure Blob Storage container

    • List Azure Blob Storage container names

    • Create an Azure Blob Storage container

    • Monitor a Blob in a Storage container

  • File Transfer can be triggered by a third-party application using the Universal Automation Center RESTful Webservice API: REST API

  • Universal Task for Azure Blob Storage can be integrated into any existing scheduling workflow in the same way as any standard Linux or Windows Task type.

  • Security is ensured by using the HTTPS protocol with support for an optional Proxy Server.

  • Support for Azure Token based Shared Access Signatures ( SAS ), Azure Account Keys and Service Principal Authentication

  • No Universal Agent needs to be installed on the Azure Cloud – the communication goes via HTTPS

 

Version Information

Template Name

Version

Template Name

Version

CS Azure Blob Storage

1.5.0

Refer to Changelog for version history information.

Software Requirements

Software Requirements for Universal Agent

  • Universal Agent for Linux or Windows Version 7.2.0.0 or later are required

  • The Universal Agent needs to be installed with python option (--python yes)

Software Requirements for Universal Controller

  • Universal Controller 7.2.0.0. or later is required

Software Requirements for the Application to be Scheduled

The Universal Task has been tested for the python Azure Storage SDK: azure-storage-blob version: 12.7.1

Key Features

The solution supports the following file transfer scenarios:

Feature

Description

Upload a file(s) to a container (Copy or Move)

  • One or more files can be uploaded to a container.

  • Move or copy are supported for the upload

  • Unix filename pattern matching support; for example, wild card support “*” to upload multiple files

  • One or more files can be Uploaded to a folder using a prefix

  • It can be decided via the Upload Write Options to:

    • Overwrite an existing object (Replace existing Object)

    • Cancel the operations in case an object with a similar name exists (Do not overwrite existing Object)

    • Add a timestamp to the uploaded Object (Timestamp) 

Download of file(s) from a container (Copy or Move)

  • One or more files should be downloaded from a bucket.

  • Move or copy must be supported for the download

  • Unix filename pattern matching support; for example, wild card support “*” to download multiple files

  • Download to a specific folder is supported

  • It can be decided via the Download Write Options to:

    • Overwrite an existing file (Replace existing File)

    • Cancel the operations in case a File with a similar name exists (Do not overwrite existing File)

    • Add a timestamp to the uploaded file (Timestamp) 

    • Perform the default Windows behaviour for copying files (Default Windows behaviour)

      • If a file with a similar name exists, the file names that are similar will be edited so that the files you copied have a number appended at the end of them. For example, if you’re copying a file named image.png to a folder that already has a file named image.png in it, the copied file will be named image (1).png.

Copy object to a container

  • An object can be copied from one container to another.

  • Folders are supported

Note: wild cards are not supported for this method.

List objects in a container

  • Show all Objects in a container

  • Show all Objects in a folder

  • Unix filename pattern matching support e.g. wild card support “*” to narrow down the objects to display

Delete object(s) in a container

  • Delete one or multiple objects in a container

  • Delete one or multiple objects in folder in a container

  • Unix filename pattern matching support e.g. wild card support “*” to narrow down the objects to be deleted

List buckets

List all container in an Azure account

Create a container

Create a new container

Monitor Blob

Monitor a Blob in a Storage container

Proxy Server Connection

A proxy server connection with or without basic authentication can be configured

Integration into 3rd Party Applications

An Azure Blob Storage file transfer can be triggered via the Universal Automation Center RESTful Web Service API within an application.

Self-Service through Web-Client

The Azure Blob Storage Task can be fully configured, monitored and updated via the Universal Controller Web-GUI

Authentication Methods

Shared Access Signatures ( SAS ), Azure Account Keys and Service Principal Authentication

Import Universal Template

To use the Universal Template, you first must perform the following steps.

  1. This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.

  2. To import the Universal Template into your Controller, follow these instructions.

  3. When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list.

Modifications of this integration, applied by users or customers, before or after import, might affect the supportability of this integration. For more information refer to Integration Modifications.

Configure Azure Blob Storage Universal Tasks

For the new Universal Task type, create a new task and enter the task-specific Details that were created in the Universal Template.

Field Descriptions for Azure Blob Storage Universal Task - Action

The Azure Blob Storage Task provides multiple different file transfer actions. For each action the specific fields are described.

Azure Blob Storage Container is abbreviated to Container in the following.

Create Container - Action

The Action Creates a new Container.

Field

Description

Action

Create-container action

Useproxy ( default is NO )

[NO | YES]

If set to YES, the fields to set-up the proxy server connections are displayed:

  • Proxy Server IP or hostname

  • Proxy Server Port

  • Proxy Server Credentials (optional)

Loglevel ( default is INFO )

Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]

Connection Type

[Azure Account Key | SAS Token | Service Principal Authentication]

If set to "Azure Account Key", the credential field Azure Account will appear for configuration.

If set to "SAS Token", the credential field SAS Token will appear for configuration.

If set to "Service Principal Authentication" the following fields will appear:

  • Credential field: Azure Client Id & Secret

  • Text filed: Azure Tenant Id

  • Text field: Storage Account Name

For Service Principal setup details, see Service Principal Authentication

Container

Name of the Container to create

Example for Azure Blob Storage Universal Tasks - Create Container

The following Task creates the Container: stonebranchpm2

List Container - Action

The Action list all container of an Azure Account.

Field

Description

Action

List-container action

Connection Type

[Azure Account Key | SAS Token | Service Principal Authentication]

If set to "Azure Account Key", the credential field Azure Account will appear for configuration.

If set to "SAS Token", the credential field SAS Token will appear for configuration.

If set to "Service Principal Authentication" the following fields will appear:

  • Credential field: Azure Client Id & Secret

  • Text filed: Azure Tenant Id

  • Text field: Storage Account Name

For Service Principal setup details, see Service Principal Authentication

Useproxy ( default is NO )

[NO | YES]

If set to YES, the fields to set-up the proxy server connections are displayed:

  • Proxy Server IP or hostname

  • Proxy Server Port

  • Proxy Server Credentials (optional)

Loglevel ( default is INFO )

Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]

Example for Azure Blob Storage Universal Tasks - List Containers

The following Task list all container of the selected Azure Account.

Upload File - Action

The Action is used to upload a single or multiple files from a Windows or Linux server to an container or a folder in a container.

Field

Description

Action

Upload-file action

Connection Type

[Azure Account Key | SAS Token | Service Principal Authentication]

If set to "Azure Account Key", the credential field Azure Account will appear for configuration.

If set to "SAS Token", the credential field SAS Token will appear for configuration.

If set to "Service Principal Authentication" the following fields will appear:

  • Credential field: Azure Client Id & Secret

  • Text filed: Azure Tenant Id

  • Text field: Storage Account Name

For Service Principal setup details, see Service Principal Authentication

Useproxy ( default is NO )

[NO | YES]

If set to YES, the fields to set-up the proxy server connections are displayed:

  • Proxy Server IP or hostname

  • Proxy Server Port

  • Proxy Server Credentials (optional)

Loglevel ( default is INFO )

Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]

Sourcefile

Source file(s) to be uploaded to a bucket or specific folder in a container.

Unix filename pattern are supported to upload a selection of files.

Supported wildcards are:

  • ? matches any single character

  • [seq] matches any character in seq

  • [!seq] matches any character not in seq

  • ”*” matches everything

Operation

[copy | move]

In case of a “move” the source file(s) is/are deleted after the upload to the container.

Prefix

A folder in a container is called prefix. In the Prefix field, only the name of the folder needs to be provided (no “/” after the folder name).

Upload Write Options

Upload Write Options

[Do not overwrite existing Object | Timestamp]

  • Do not overwrite existing Object: Cancel the operations in case an object with a similar name exists

  • Timestamp: Add a timestamp to the uploaded Object

Example for Azure Blob Storage Universal Tasks - Upload File

The following Task uploads from the Linux directory the files: /home/stonebranch/demo/azure/out/test[1-4]* to the the container stonebranchpm, folder incoming.

List Objects - Action

The Action is used to display objects in a Azure Container or a specific folder in an Azure Container (prefix).

Field

Description

Action

List-objects action

Connection Type

[Azure Account Key | SAS Token | Service Principal Authentication]

If set to "Azure Account Key", the credential field Azure Account will appear for configuration.

If set to "SAS Token", the credential field SAS Token will appear for configuration.

If set to "Service Principal Authentication" the following fields will appear:

  • Credential field: Azure Client Id & Secret

  • Text filed: Azure Tenant Id

  • Text field: Storage Account Name

For Service Principal setup details, see Service Principal Authentication

Useproxy ( default is NO )

[NO | YES]

If set to YES, the fields to set-up the proxy server connections are displayed:

  • Proxy Server IP or hostname

  • Proxy Server Port

  • Proxy Server Credentials (optional)

Loglevel ( default is INFO )

Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]

Container

Container in which the objects should be listed

Prefix

A folder in a container is called prefix. In the Prefix field, only the name of the folder needs to be provided (no “/” after the folder name).

If a prefix is provided, only objects in the folder with the prefix name are listed in the output.

Blob

Objects matching the given Blob are listed.

Unix filename pattern are supported to list only a selection of files:
Supported wildcards are:

  • ? matches any single character

  • [seq] matches any character in seq

  • [!seq] matches any character not in seq

  • ”*” matches everything

Example:

Blob = test* : matches everything starting with test

Blob = test[1-2].txt : matches test1.txt, test2.txt

Blob = test[!1].txt: does not match test1.txt

Blob = test?.txt: matches test1.txt, test2.txt etc.

Show Details

Show details like creation timestamp in the output.

Example for Azure Blob Storage Universal Tasks - List Objects

The following Task list all objects matching the criteria test* in the container stonebranchpm in the folder incoming.

Download File - Action

This Action downloads one or multiple files from an Azure Container to a Linux or Windows folder.

Field

Description

Action

Download-file action

Connection Type

[Azure Account Key | SAS Token | Service Principal Authentication]

If set to "Azure Account Key", the credential field Azure Account will appear for configuration.

If set to "SAS Token", the credential field SAS Token will appear for configuration.

If set to "Service Principal Authentication" the following fields will appear:

  • Credential field: Azure Client Id & Secret

  • Text filed: Azure Tenant Id

  • Text field: Storage Account Name

For Service Principal setup details, see Service Principal Authentication

Useproxy ( default is NO )

[NO | YES]

If set to YES, the fields to set-up the proxy server connections are displayed:

  • Proxy Server IP or hostname

  • Proxy Server Port

  • Proxy Server Credentials (optional)

Loglevel ( default is INFO )

Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]

Target Directory

Linux or Windows Target Directory

For example, C:\tmp\ or /home/ubuntu/download

Unix filename pattern are supported to download a selection of files.

Supported wildcards are:

  • ? matches any single character

  • [seq] matches any character in seq

  • [!seq] matches any character not in seq

  • ”*” matches everything

Operation

[copy | move]

In case of a “move” the objects are deleted after they have been download from the container.

Download Write Options

Download Write Options:

[Replace existing File | Do not overwrite existing File | Timestamp | Default Windows behaviour]

  • Replace existing File: overwrite an existing file

  • Do not overwrite existing File: cancel the operations in case a File with a similar name exists

  • Timestamp: add a timestamp to the uploaded file

  • Default Windows behaviour: perform the default Windows behaviour for copying files. If a file with a similar name exists, the file names that are similar will be edited so that the files you copied have a number appended at the end of them. For example, if you are copying a file named image.png to a folder that already has a file named image.png in it, the copied file will be named image (1).png.

Example for Azure Blob Storage Universal Tasks - Download File

The following Task downloads from the container stonebranchpm all blobs matching the criteria test[1-2]* to the Linux directory: /home/stonebranch/demo/azure/in

Delete Objects - Action

The Actions is used to delete an object in an Azure Container or folder

Field

Description

Action

Delete-objects action