Panel | |
---|---|
|
...
Storing data in the cloud becomes an integral part of most modern IT landscapes. With Universal Automation Center, you can securely automate your AWS, Azure, Google and MinIO File Transfers and integrate them into your existing scheduling flows.
...
The following file transfer commands are supported:
Upload a file(s) to an S3 bucket
Download of file(s) from an S3 bucket
Transfer files between S3 buckets
List objects in an S3 bucket
Delete object(s) in an S3 bucket
List S3 bucket names
Create an S3 bucket
- Support for AWS S3 prefixes to simulate a folder structure and to improve performance
Monitor for a key(s) in a bucket.
File Transfer can be trigger by a third-party application using the Universal Automation Center RESTfull Webservice API: REST API.
Universal Task for AWS S3 can be integrated into any existing scheduling workflow in the same way as any standard Linux or Windows Task type.
Security is ensured by using the HTTPS protocol with support for an optional Proxy Server.
AWS IAM Role Based Access ( RBCA ) is supported.
AWS canned ACLs are supported; for example, to grant full access to the bucket owner.
No Universal Agent needs to be installed on the AWS Cloud – the communication goes via HTTPS.
...
Template Name | Version |
---|---|
AWS-S3 | 1.56.250 |
Refer to
...
Changelog for version history information.
Software Requirements
Software Requirements Universal Agents and Universal Controller
Universal Agent for Linux or Windows Version 7.20.0.0 or later are required.
The Universal Agent needs to be installed with python option (--python yes).
Software Requirements Universal Controller
Universal Controller 7.20.0.0. or later is required.
Software Requirements for the Application to be scheduled
...
Name | Description |
Upload a file(s) to a bucket (Copy or Move) |
|
Download of file(s) from a bucket (Copy or Move) |
|
Copy object to a bucket |
|
List objects in a bucket |
|
Delete object(s) in a bucket |
|
List buckets | List all bucket in an AWS S3 account |
Create an S3 bucket | Create a new S3 bucket |
Delete an S3 bucket | Delete an S3 bucket |
Monitor Key | Monitor if a certain key(s) exists in an S3 bucket. Unix filename pattern matching support; for example, wild card support “*” to narrow down the keys to be monitored. |
Proxy Server Connection | A proxy server connection with or without basic authentication can be configured |
Integration into 3rd Party Applications | An AWS S3 file transfer can be triggered via the Universal Automation Center RESTfull Webservice API within an application. |
Self-Service through Web-Client | The S3 Task can be fully configured, monitored and updated via the Universal Controller Web-GUI |
...
This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.
In the Universal Controller UI, select Administration >Configuration > Universal Templates to display the current list of Universal Templates.
Click Import Template.
Select the template ZIP file and Import.
...
The Amazon S3 Task provides multiple different file transfer actions. For each action the specific fields are described.
Create Bucket - Action
This action creates The Action Creates a new S3 bucket.
Field | Description |
Action | create-bucket action |
AWS_ACCESS_KEY_ID | AWS Account credentials Access Key |
AWS_DEFAULT_REGION | AWS Account credentials Region |
AWS_SECRET_ACCESS_KEY | AWS Account credentials Secret Access Key |
Useproxy ( default is NO ) | [NO | YES] If set to YES, the fields to set-up the proxy server connections are displayed:
|
Loglevel ( default is INFO ) | Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL] |
Role Based Access | [NO | YES] If set to YES, the fields to set-up the IAM Role Based Access Connection are displayed:
STS: AWS Security Token Service is used to create and provide trusted users with temporary security credentials that can control access to your AWS resources. |
Endpoint URL | Only used in case of a MinIO server; connection Endpoint URL for the MinIO storage system |
Bucket | Name of the AWS Bucket to create |
Example for AWS S3 Universal Tasks - Create Bucket
List Buckets - Action
This action lists The Action list all buckets of an AWS Account.
...
Example for Amazon S3 Universal Tasks - List Buckets
Upload File - Action
This action The Action is used to upload a single or multiple files from a Windows or Linux server to an AWS bucket or a folder in an AWS bucket.
Field | Description |
Action | Upload-file action |
AWS_ACCESS_KEY_ID | AWS Account credentials Access Key |
AWS_DEFAULT_REGION | AWS Account credentials Region |
AWS_SECRET_ACCESS_KEY | AWS Account credentials Secret Access Key |
Useproxy ( default is NO ) | [NO | YES] If set to YES, the fields to set-up the proxy server connections are displayed:
|
Loglevel ( default is INFO ) | Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL] |
Role Based Access | [NO | YES] If set to YES, the fields to set-up the IAM Role Based Access Connection are displayed:
STS: AWS Security Token Service is used to create and provide trusted users with temporary security credentials that can control access to your AWS resources. |
ACL | bucket-owner-full-control | private | public-read | public-read-write | aws-exec-read | authenticated-read | bucket-owner-read | log-delivery-write AWS canned ACLs; for details, refer to: The default ACL bucket-owner-full-control grant full access to the bucket owner. |
Endpoint URL | Only used in case of a MinIO server; connection Endpoint URL for the MinIO storage |
Sourcefile | Source file(s) to be uploaded to a bucket or specific folder in a bucket. Unix filename pattern are supported to upload a selection of files. Supported wildcards are:
|
Operation | [copy | move] In case of a “move” the source file(s) is/are deleted after the upload to the bucket. |
Prefix | A folder in a bucket is called prefix in AWS. In the field Prefix only the name of the folder needs to be provided no “/” after the folder name. |
Upload Write Options | Upload Write Options [Replace existing Object | Do not overwrite existing Object | Timestamp]
|
Example for Amazon S3 Universal Tasks - Upload File
List Objects - Action
This action The Action is used to display objects in a bucket or a specific bucket folder (prefix).
...
Download File - Action
This Action is used to download downloads one or multiple files from a bucket to a Linux or Windows folder.
Field | Description |
Action | Download-file action |
AWS_ACCESS_KEY_ID | AWS Account credentials Access Key |
AWS_DEFAULT_REGION | AWS Account credentials Region |
AWS_SECRET_ACCESS_KEY | AWS Account credentials Secret Access Key |
Useproxy ( default is NO ) | [NO | YES] If set to YES, the fields to set-up the proxy server connections are displayed:
|
Loglevel ( default is INFO ) | Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL] |
Role Based Access | [NO | YES] If set to YES, the fields to set-up the IAM Role Based Access Connection are displayed:
STS: AWS Security Token Service is used to create and provide trusted users with temporary security credentials that can control access to your AWS resources. |
Endpoint URL | Only used in case of a MinIO server; connection Endpoint URL for the MinIO storage system |
Target Directory | Linux or Windows Target Directory For example, C:\tmp\ or /home/ubuntu/download Unix filename pattern are supported to download a selection of files. Supported wildcards are:
|
Operation | [copy | move] In case of a “move” the objects are deleted after they have been download from the bucket. |
Source Prefix | A folder in a bucket is called prefix in AWS. In the field Prefix only the name of the folder needs to be provided no “/” after the folder name. If a prefix is provided only objects in the folder with the prefix name are select for the action. |
Operation | [copy | move] In case of a “move” the objects are deleted after they have been download from the bucket. |
Download Write Options | Download Write Options: [Replace existing File | Do not overwrite existing File | Timestamp | Default Windows behaviour]
|
Example for Amazon S3 Universal Tasks - Download File
Delete Objects - Action
This action The Actions is used to delete an object in a bucket or folder.
...
Copy Object to Bucket - Action
This action Action is used to copy a single or multiple objects from one bucket to another bucket in AWS.
Field | Description | ||||||||||
Action | Copy-object-to-bucket action | ||||||||||
AWS_ACCESS_KEY_ID | AWS Account credentials Access Key | ||||||||||
AWS_DEFAULT_REGION | AWS Account credentials Region | ||||||||||
AWS_SECRET_ACCESS_KEY | AWS Account credentials Secret Access Key | ||||||||||
Useproxy ( default is NO ) | [NO | YES] If set to YES, the fields to set-up the proxy server connections are displayed:
| ||||||||||
Loglevel ( default is INFO ) | Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL] | ||||||||||
Role Based Access | [NO | YES] If set to YES, the fields to set-up the IAM Role Based Access Connection are displayed:
STS: AWS Security Token Service is used to create and provide trusted users with temporary security credentials that can control access to your AWS resources. | ||||||||||
ACL | bucket-owner-full-control | private | public-read | public-read-write | aws-exec-read | authenticated-read | bucket-owner-read | log-delivery-write AWS canned ACLs; for details, refer to: The default ACL bucket-owner-full-control grant full access to the bucket owner. | ||||||||||
Endpoint URL | Only used in case of a MinIO server; connection Endpoint URL for the MinIO storage system | ||||||||||
Bucket | AWS bucket name | Target Bucket | target Bucket, where the object For example: sbtransfer | ||||||||
Source Prefix | A folder in a bucket is called prefix in AWS. In the field Prefix only the name of the folder needs to be provided no “/” after the folder name. If a prefix is provided only objects in the folder with the prefix name are select for the action. For example: out/reports | ||||||||||
Target Bucket | target Bucket, where the object(s) will be copied to | ||||||||||
S3key | File to be copied from one bucket to another | ||||||||||
TargetS3Key | Target file name. The TargetS3key supports pattern match. For example:
This copies all files from bucket stonebranchpm in sbtransfer in the directory dir1 starting out/reports starting with the file name report to the directory dir2in/ in reports in the bucket stonebranchpm1sbtransfer2.
| ||||||||||
Target Prefix | A folder in a bucket is called prefix in AWS. In the field Prefix only the name of the folder needs to be provided no “/” after the folder name. If a prefix is provided only objects in the folder with the prefix name are select for the action. For example: in/reports | ||||||||||
Upload Write Options | Upload Write Options [Replace existing Object | Do not overwrite existing Object | Timestamp]
|
Example for Amazon S3 Universal Tasks - Copy Object to Bucket
Delete Bucket - Action
This action is used to delete a bucket.
Field | Description |
Action | Delete-bucket action |
AWS_ACCESS_KEY_ID | AWS Account credentials Access Key |
AWS_DEFAULT_REGION | AWS Account credentials Region |
AWS_SECRET_ACCESS_KEY | AWS Account credentials Secret Access Key |
Useproxy ( default is NO ) | [NO | YES] If set to YES, the fields to set-up the proxy server connections are displayed:
|
Loglevel ( default is INFO ) | Universal Task logging settings [DEBUG | INFO| WARNING | ERROR | CRITICAL] |
Role Based Access | [NO | YES] If set to YES, the fields to set-up the IAM Role Based Access Connection are displayed:
STS: AWS Security Token Service is used to create and provide trusted users with temporary security credentials that can control access to your AWS resources. |
Endpoint URL | Only used in case of a MinIO server; connection Endpoint URL for the MinIO storage system |
Bucket | Name of the bucket to be deleted |
...
Example for Amazon S3 Universal Tasks - Monitor Key
Anchor | ||||
---|---|---|---|---|
|
Changelog
ut-cs-aws-s3-1.6.0 (2024-08-14)
Enhancements
- Support for AWS S3 prefixes to simulate a folder structure and to improve performance
ut-cs-aws-s3-1.5.25 (2024-05-09)
Enhancements
- Proxy setting for https adjusted to support python 3.11
ut-cs-aws-s3-1.0.0
- Initial Version
...