Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Panel

Table of Contents

...

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at https://www.stonebranch.com/integration-hub/Terms-and-Privacy/Terms-of-Use/

Version Information


Template NameVersion
AWS-S31.6.0

Refer to Changelog for version history information.

Introduction

Storing data in the cloud becomes an integral part of most modern IT landscapes. With Universal Automation Center, you can securely automate your AWS, Azure, Google and MinIO File Transfers and integrate them into your existing scheduling flows.

...

  • The following file transfer commands are supported:

    • Upload a file(s) to an S3 bucket

    • Download of file(s) from an S3 bucket

    • Transfer files between S3 buckets

    • List objects in an S3 bucket

    • Delete object(s) in an S3 bucket

    • List S3 bucket names

    • Create an S3 bucket

  • Support for AWS S3 prefixes to simulate a folder structure and to improve performance
  • Monitor for a key(s) in a bucket.

  • File Transfer can be trigger by a third-party application using the Universal Automation Center RESTfull Webservice API: REST API.

  • Universal Task for AWS S3 can be integrated into any existing scheduling workflow in the same way as any standard Linux or Windows Task type.

  • Security is ensured by using the HTTPS protocol with support for an optional Proxy Server.

  • AWS IAM Role Based Access ( RBCA ) is supported.

  • AWS canned ACLs are supported; for example, to grant full access to the bucket owner.

  • No Universal Agent needs to be installed on the AWS Cloud – the communication goes via HTTPS.

Version Information

...

Refer to Changelog for version history information.

Software Requirements

...

Software Requirements

Software Requirements Universal Agents and Universal Controller

  • Universal Agent for Linux or Windows Version 7.20.0.0 or later are required.

  • The Universal Agent needs to be installed with python option (--python yes).

Software Requirements Universal Controller

  • Universal Controller 7.20.0.0. or later is required.

Software Requirements for the Application to be scheduled

...

NameDescription

Upload a file(s) to a bucket (Copy or Move)

  • A single or multiple files can be uploaded to a bucket.

  • Move or copy are supported for the upload

  • Unix filename pattern matching support e.g. wild card support “*” to upload multiple files

  • Prefixes are supported to simulate a folder structure
  • A single or multiple files can be Uploaded to a folder using a prefix

  • It can be decided via an the Upload Write Options:

    • to overwrite an existing object (Replace existing Object)

    • to cancel the operations in case an object with a similar name exists (Do not overwrite existing Object)

    • to add a timestamp to the uploaded Object (Timestamp

Download of file(s) from a bucket (Copy or Move)

  • A single or multiple files should be downloaded from a bucket.

  • Move or copy must be supported for the download

  • Unix filename pattern matching support e.g. wild card support “*” to download multiple files

  • Prefixes are supported to simulate a folder structure
  • Download to a specific folder is supported

  • It can be decided via an the Download Write Options:

    • to overwrite an existing file (Replace existing File)

    • to cancel the operations in case a File with a similar name exists (Do not overwrite existing File)

    • to add a timestamp to the uploaded file (Timestamp

    • to perform the default Windows behaviour for copying files (Default Windows behaviour)

      • If a file with a similar name exists, the file names that are similar will be edited so that the files you copied have a number appended at the end of them. For example, if you’re copying a file named image.png to a folder that already has a file named image.png in it, the copied file will be named image (1).png.

Copy object to a bucket

  • An object can be copied from one AWS S3 bucket to another.

  • Folders are supportPrefixes are supported to simulate a folder structure

List objects in a bucket

  • Show all Objects in a bucket

  • Show all Objects in a folder

  • Unix filename pattern matching support e.g. wild card support “*” to narrow down the objects to display

Delete object(s) in a bucket

  • Delete one or multiple objects in a bucket

  • Delete one or multiple objects in folder in a bucket

  • Unix filename pattern matching support; for example, wild card support “*” to narrow down the objects to be deleted

List buckets

List all bucket in an AWS S3 account

Create an S3 bucket

Create a new S3 bucket

Delete an S3 bucket

Delete an S3 bucket

Monitor Key

Monitor if a certain key(s) exists in an S3 bucket.

Unix filename pattern matching support; for example, wild card support “*” to narrow down the keys to be monitored.

Proxy Server Connection

A proxy server connection with or without basic authentication can be configured

Integration into 3rd Party Applications

An AWS S3 file transfer can be triggered via the Universal Automation Center RESTfull Webservice API within an application.

Self-Service through Web-Client

The S3 Task can be fully configured, monitored and updated via the Universal Controller Web-GUI

...

  1. This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.

  2. In the Universal Controller UI, select Administration >Configuration > Universal Templates to display the current list of Universal Templates.

  3. Click Import Template.

  4. Select the template ZIP file and Import.

...

The Amazon S3 Task provides multiple different file transfer actions. For each action the specific fields are described.

Create Bucket - Action

This action creates The Action Creates a new S3 bucket.

FieldDescription

Action

create-bucket action

AWS_ACCESS_KEY_ID

AWS Account credentials Access Key

AWS_DEFAULT_REGION

AWS Account credentials Region

AWS_SECRET_ACCESS_KEY

AWS Account credentials Secret Access Key

Useproxy ( default is NO )

[NO | YES]

If set to YES, the fields to set-up the proxy server connections are displayed:

  • Proxy Server IP or hostname

  • Proxy Server Port

  • Proxy Server Credentials (optional)

Loglevel ( default is INFO )

Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]

Role Based Access

[NO | YES]

If set to YES, the fields to set-up the IAM Role Based Access Connection are displayed:

  • Role Arn: Amazon Role, which is applied for the connection; for example, rn:aws:iam::111222333444:role/SB-AWSS3

  • Service Name [STS | S3 ] , default is STS

STS: AWS Security Token Service is used to create and provide trusted users with temporary security credentials that can control access to your AWS resources.

Endpoint URL

Only used in case of a MinIO server; connection Endpoint URL for the MinIO storage system

Bucket

Name of the AWS Bucket to create

Example for AWS S3 Universal Tasks - Create Bucket

List Buckets - Action

This action lists The Action list all buckets of an AWS Account.

...

Example for Amazon S3 Universal Tasks - List Buckets

Upload File - Action

This action The Action is used to upload a single or multiple files from a Windows or Linux server to an AWS bucket or a folder in an AWS bucket.

FieldDescription

Action

Upload-file action

AWS_ACCESS_KEY_ID

AWS Account credentials Access Key

AWS_DEFAULT_REGION

AWS Account credentials Region

AWS_SECRET_ACCESS_KEY

AWS Account credentials Secret Access Key

Useproxy ( default is NO )

[NO | YES]

If set to YES, the fields to set-up the proxy server connections are displayed:

  • Proxy Server IP or hostname

  • Proxy Server Port

  • Proxy Server Credentials (optional)

Loglevel ( default is INFO )

Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]

Role Based Access

[NO | YES]

If set to YES, the fields to set-up the IAM Role Based Access Connection are displayed:

  • Role Arn: Amazon Role, which is applied for the connection e.g. rn:aws:iam::111222333444:role/SB-AWSS3

  • Service Name [STS | S3 ] , default is STS

STS: AWS Security Token Service is used to create and provide trusted users with temporary security credentials that can control access to your AWS resources.

ACL

bucket-owner-full-control | private | public-read | public-read-write | aws-exec-read | authenticated-read | bucket-owner-read | log-delivery-write

AWS canned ACLs; for details, refer to:

AWS canned ACL

The default ACL bucket-owner-full-control grant full access to the bucket owner.

Endpoint URL

Only used in case of a MinIO server; connection Endpoint URL for the MinIO storage

Sourcefile

Source file(s) to be uploaded to a bucket or specific folder in a bucket.

Unix filename pattern are supported to upload a selection of files.

Supported wildcards are:

  • ? matches any single character

  • [seq] matches any character in seq

  • [!seq] matches any character not in seq

  • ”*” matches everything

Operation

[copy | move]

In case of a “move” the source file(s) is/are deleted after the upload to the bucket.

Prefix

A folder in a bucket is called prefix in AWS. In the field Prefix only the name of the folder needs to be provided no “/” after the folder name.

Upload Write Options

Upload Write Options

[Replace existing Object | Do not overwrite existing Object | Timestamp]

  • Replace existing Object: Overwrite an existing object

  • Do not overwrite existing Object: Cancel the operations in case an object with a similar name exists

    • Timestamp: Add a timestamp to the uploaded Object

Example for Amazon S3 Universal Tasks - Upload File

List Objects - Action

This action The Action is used to display objects in a bucket or a specific bucket folder (prefix).

...

Download File - Action

This Action is used to download downloads one or multiple files from a bucket to a Linux or Windows folder.

FieldDescription

Action

Download-file action

AWS_ACCESS_KEY_ID

AWS Account credentials Access Key

AWS_DEFAULT_REGION

AWS Account credentials Region

AWS_SECRET_ACCESS_KEY

AWS Account credentials Secret Access Key

Useproxy ( default is NO )

[NO | YES]

If set to YES, the fields to set-up the proxy server connections are displayed:

  • Proxy Server IP or hostname

  • Proxy Server Port

  • Proxy Server Credentials (optional)

Loglevel ( default is INFO )

Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]

Role Based Access

[NO | YES]

If set to YES, the fields to set-up the IAM Role Based Access Connection are displayed:

  • Role Arn: Amazon Role, which is applied for the connection e.g. rn:aws:iam::111222333444:role/SB-AWSS3

  • Service Name [STS | S3 ] , default is STS

STS: AWS Security Token Service is used to create and provide trusted users with temporary security credentials that can control access to your AWS resources.

Endpoint URL

Only used in case of a MinIO server; connection Endpoint URL for the MinIO storage system

Target Directory

Linux or Windows Target Directory

For example, C:\tmp\ or /home/ubuntu/download

Unix filename pattern are supported to download a selection of files.

Supported wildcards are:

  • ? matches any single character

  • [seq] matches any character in seq

  • [!seq] matches any character not in seq

  • ”*” matches everything

Source Prefix

A folder in a bucket is called prefix in AWS. In the field Prefix only the name of the folder needs to be provided no “/” after the folder name.

If a prefix is provided only objects in the folder with the prefix name are select for the action.

Operation

[copy | move]

In case of a “move” the objects are deleted after they have been download from the bucket.

Download Write Options

Download Write Options:

[Replace existing File | Do not overwrite existing File | Timestamp | Default Windows behaviour]

  • Replace existing File : overwrite an existing file

  • Do not overwrite existing File: cancel the operations in case a File with a similar name exists

  • Timestamp: add a timestamp to the uploaded file

  • Default Windows behaviour: perform the default Windows behaviour for copying files. If a file with a similar  name exists, the file names that are similar will be edited so that the files you copied have a number appended at the end of them. For example, if you’re copying a file named image.png to a folder that already has a file named image.png in it, the copied file will be named image (1).png.

Example for Amazon S3 Universal Tasks - Download File

Image RemovedImage Added

Delete Objects - Action

This action The Actions is used to delete an object in a bucket or folder.

...

Copy Object to Bucket - Action

This action Action is used to copy a single or multiple objects from one bucket to another bucket in AWS.

Target file name.

The TargetS3key supports pattern match. For example:

Bucket: stonebranchpm

Target Bucket: stonebranchpm1

S3key: dir1/report* ;

Target s3key: dir2/*
FieldDescription

Action

Copy-object-to-bucket action

AWS_ACCESS_KEY_ID

AWS Account credentials Access Key

AWS_DEFAULT_REGION

AWS Account credentials Region

AWS_SECRET_ACCESS_KEY

AWS Account credentials Secret Access Key

Useproxy ( default is NO )

[NO | YES]

If set to YES, the fields to set-up the proxy server connections are displayed:

  • Proxy Server IP or hostname

  • Proxy Server Port

  • Proxy Server Credentials (optional)

Loglevel ( default is INFO )

Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]

Role Based Access

[NO | YES]

If set to YES, the fields to set-up the IAM Role Based Access Connection are displayed:

  • Role Arn: Amazon Role, which is applied for the connection e.g. rn:aws:iam::111222333444:role/SB-AWSS3

  • Service Name [STS | S3 ] , default is STS

STS: AWS Security Token Service is used to create and provide trusted users with temporary security credentials that can control access to your AWS resources.

ACL

bucket-owner-full-control | private | public-read | public-read-write | aws-exec-read | authenticated-read | bucket-owner-read | log-delivery-write

AWS canned ACLs; for details, refer to:

AWS canned ACL

The default ACL bucket-owner-full-control grant full access to the bucket owner.

Endpoint URL

Only used in case of a MinIO server; connection Endpoint URL for the MinIO storage system

Bucket

TargetS3Key

AWS bucket name

Target Bucket

target Bucket, where the object(s) will be copied to

S3key

File to be copied from one bucket to another

For example: sbtransfer

Source Prefix

A folder in a bucket is called prefix in AWS. In the field Prefix only the name of the folder needs to be provided no “/” after the folder name.

If a prefix is provided only objects in the folder with the prefix name are select for the action.

For example: out/reports

Target Bucket

target Bucket, where the object(s) will be copied to

S3key

File to be copied from one bucket to another

TargetS3Key

Target file name.

For example:

  • Source Bucket: sbtransfer
  • Source Prefix: out/reports
  • S3key: report*
  • Target Bucket: sbtransfer2
  • Target Prefix: in/reports
  • Target s3key: empty

This copies all files from bucket stonebranchpm in sbtransfer in the directory dir1 starting out/reports starting with the file name report to the directory dir2in/ in reports in the bucket stonebranchpm1sbtransfer2.

Note
titleNote

If the field TargetS3key is left blank, is left blank and no target prefix is provided the entire source key - including the directory - will be copied to the target bucket. For example:

Bucket: stonebranchpmTarget Bucket

Source Prefix: stonebranchpm1 dir1

S3key: dir1/ report* ;

Target Bucket: stonebranchpm1

Target Prefix: empty

Target s3key: empty

This copies dir1/all files starting with report* from bucket stonebranchpm to the bucket stonebranchpm1the bucket stonebranchpm1 incl. the source dir name "dir1"

For example:

The report1.txt and report2.txt are located in the source dir: dir1

The action will copy the files report1.txt and report2.txt to the bucket stonebranchpm1 directory dir1. If dir1 does not exit, it will be created.


Note
titleATTENTION

Do not use a non-pattern match Target S3key when the S3key uses pattern match.  For example:

Bucket: stonebranchpm

Target Bucket: stonebranchpm1

S3key: dir1/report* ;

Target s3key: dir2/report1.txttx

This set-up will copy only one file, even if multiple files will match the S3Key in the source.only one file, even if multiple files will match the S3Key in the source.


Target Prefix

A folder in a bucket is called prefix in AWS. In the field Prefix only the name of the folder needs to be provided no “/” after the folder name.

If a prefix is provided only objects in the folder with the prefix name are select for the action.

For example: in/reports

Upload Write Options

Upload Write Options

[Replace existing Object | Do not overwrite existing Object | Timestamp]

  • Replace existing Object: Overwrite an existing object

  • Do not overwrite existing Object: Cancel the operations in case an object with a similar name exists

  • Timestamp: Add a timestamp to the uploaded Object

Example for Amazon S3 Universal Tasks - Copy Object to Bucket

Image RemovedImage Added

Delete Bucket - Action

This action is used to delete a bucket.

Field

Description

Action

Delete-bucket action

AWS_ACCESS_KEY_ID

AWS Account credentials Access Key

AWS_DEFAULT_REGION

AWS Account credentials Region

AWS_SECRET_ACCESS_KEY

AWS Account credentials Secret Access Key

Useproxy ( default is NO )

[NO | YES]

If set to YES, the fields to set-up the proxy server connections are displayed:

  • Proxy Server IP or hostname

  • Proxy Server Port

  • Proxy Server Credentials (optional)

Loglevel ( default is INFO )

Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL]

Role Based Access

[NO | YES]

If set to YES, the fields to set-up the IAM Role Based Access Connection are displayed:

  • Role Arn: Amazon Role, which is applied for the connection e.g. rn:aws:iam::111222333444:role/SB-AWSS3

  • Service Name [STS | S3 ] , default is STS

STS: AWS Security Token Service is used to create and provide trusted users with temporary security credentials that can control access to your AWS resources.

Endpoint URL

Only used in case of a MinIO server; connection Endpoint URL for the MinIO storage system

Bucket

Name of the bucket to be deleted

...

Example for Amazon S3 Universal Tasks - Monitor Key


Anchor
Changelog
Changelog

Changelog

ut-cs-aws-s3-1.6.0 (2024-08-14)

Enhancements

  • Support for AWS S3 prefixes to simulate a folder structure and to improve performance

ut-cs-aws-s3-1.5.25 (2024-05-09)

Enhancements

  • Proxy setting for https adjusted to support python 3.11 

ut-cs-aws-s3-1.0.0

  • Initial Version

...