...
AWS Batch is a set of batch management capabilities that enables developers, scientists, and engineers to easily and efficiently run hundreds of thousands of batch computing jobs on AWS.
This Universal Extension provides tha the capability to submit new AWS Batch Jobs as well as read the status for an existing AWS Batch Job.
Version Information
Template Name | Extension Name | Extension Version |
---|---|---|
AWS Batch | ue-aws-batch | 1.3.1 |
Refer to ChangeLog for version history information.
Software Requirements
This integration requires a Universal Agent and a Python runtime to execute the Universal Task.
Software Requirements for Universal Template and Universal Task
Requires Python 3.7.0 or higher. Tested with the Universal Agent bundled Python distribution.
...
- Universal Agent for Windows x64 Version 7.02.0.0 and later with python options installed.
- Universal Agent for Linux Version 7.02.0.0 and later with python options installed.
...
Universal Controller Version 7.02.0.0 and later.
Network and Connectivity Requirements
Extensions' Univesal Universal Agent host should be able to reach AWS Batch REST endpoints, through the configured VPC.
More information on setting up AWS Batch VPC, can be found in the AWS Batch official user guide.
Key Features
This Universal Extension provides the following key features:
- Support to submit a new Batch Job, with option to Terminate Job after a timeout period.
- Support to read Batch Job status for submit a new Batch Job and wait until it reaches state "success" or "failed".
- Support to read Batch Job status for an existing Job ID.
- Support for authorization via IAM Role-Based Access Control (RBAC) strategy.
- Support for Proxy communication via HTTP/HTTPS protocol.
...
This Universal Task requires the the Resolvable Credentials feature, check . Check that the Resolvable Credentials Permitted system property has been set to true. For more information about Resolvable Credentials click here.
Download the provided ZIP file.
In the Universal Controller UI, select Administration >Configuration > Universal Templates to display the current list of Universal Templates.
Click Import Template.
Select the template ZIP file and Import.
...
To import the Universal Template into your Controller, follow the instructions here.
When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list
...
.
Configure Universal Task
For the a new Universal Task type AWS Batch, create a new task, and enter the task-specific details that were created in the Universal Template.required input fields.
Input Fields
The input fields for this Universal Extension are described below.
Field | Input type | Default value | Type | Description | |
---|---|---|---|---|---|
Action | Required | Submit Job | Choice | The action Action performed upon the task execution. Available actions:
| |
AWS Region RequiredOptional since version 1.1.0 | Optional | - | Text | Region for the Amazon Web Service. Find more information about the AWS Service endpoints and quotas here. | |
AWS Credentials | Required | - | Credentials | The AWS account credentials. They are comprised of:
| |
Role Based Access | Optional | False | Boolean | Special type of authorization is provided by Role Assumption where the client sends his own credentials and the role he wants to assume from another user. If allowed, the client receives temporary credentials with limited time access to some resources. | |
Role ARN | Optional | - | Text | The Role Amazon Resource When AWS Region is not populated as part of the task definition, during task execution the integration will look for credentials on the task execution environment. Refer to configuration options for more information. | |
AWS Credentials Optional since version 1.1.0 | Optional | - | Credentials | The Credentials definition should be as follows.
When AWS Credentials are not populated as part of the task definition, during task execution the integration will look for AWS Credentials on the task execution environment. Refer to configuration options for more information. | |
Role Based Access | Optional | False | Boolean | Special type of authorization is provided by Role Assumption where the client sends his own credentials and the role he wants to assume from another user. If allowed, the client receives temporary credentials with limited time access to some resources. | |
Role ARN | Optional | - | Text | Role Amazon Resource Name (ARN) to have access to the SQS queue.Role ARN format: Required when Role Based Access has been checked. | |
Job Name | Optional | - | Text | The name Name of the Batch Job that will be submitted. | |
Job Definition | Optional | - | Text | The job Job definition used by this job. This value can be one of name , name:revision , or the Amazon Resource Name (ARN) for the job definition. If name is specified without a revision then the latest active revision is used. Required when Submit Job action has been selected. | |
Job Queue | Optional | - | Text | The job queue where the job is submitted. You can specify either the name or the Amazon Resource Name (ARN) of the queue. Required when Submit Job action has been selected. | |
Job Timeout | Optional | - | Integer | Can be filled when Action = Submit Job. The timeout configuration (in seconds) for this Submit Job operation. You can specify a timeout duration after which Batch terminates your jobs if they haven'thave not finished. If a job is terminated due to a timeout, it isn'tis not retried. The minimum value for the timeout is 60 seconds. This configuration overrides any timeout configuration specified in the job definition. Optional when Submit Job action has been selected. | |
Container Overrides Script Introduced in version 1.1.0 | Optional | - | Script Field | The overrides that user want to apply on the container. The script payload should be in JSON format. | |
Additional Job Parameters | Optional | - | Array | Additional parameters passed to the job that replace parameter substitution placeholders that are set in the job definition. Parameters are specified as a key and value pair mapping. Parameters in a SubmitJob request override any corresponding parameter defaults from the job definition. Optional when Submit Job action has been selected. | |
Job ID | Optional | - | Text | The Job ID of an already submitted Batch Job. Required when Read Job Status action has been selected. | |
Use Proxy | Optional | False | Boolean | Flag to indicate whether Proxy shall be used in the communication with AWS. | |
Proxy Type | Optional | HTTP | Choice | Type of proxy connection to be used. Available options are:
Required when Use Proxy is checked. | |
Proxy | Optional | - | Text | Comma separated list of Proxy servers. Valid formats: Required when Use Proxy is checked. | |
Proxy CA Bundle File | Optional | - | Text | The path to a custom certificate bundle to use when establishing SSL/TLS connections with proxy. Used when Proxy Type is configured for "HTTPS" or "HTTPS With PasswordCredentials". | |
Proxy Credentials | Optional | - | Credentials | Credentials to be used for the proxy communication. They are comprised of:
Required when Proxy Type is configured for "HTTPS" or "HTTPS With PasswordCredentials". |
Task Examples
Submit Job
Example of AWS Batch Universal Task for submitting a new AWS Batch Job.
Read Job Status
Example of AWS Batch Universal Task for reading the status of an existing Batch Job by id.
Submit Job with Role Based Access Enabled
Wait for Success or Failure Introduced in version 1.2.0 | Optional | False | Boolean | If selected, the task will continue running until Job reaches the "SUCCEDED" or "FAILED" state. Required for Action "Submit Job". |
Polling Interval Introduced in version 1.2.0 | Optional | 30 | Integer | The polling interval in seconds between checking for the Job status. Required when Wait for Success or Failure ="True". |
Show full job information on Extension Output Introduced in version 1.3.0 | Optional | False | Boolean | Controls visibility on the latest Full Job Information as provided by AWS Batch Service. |
Task Examples
Submit Job
Example of AWS Batch Universal Task for submitting a new AWS Batch Job, with Role Based Access enabled.
...
.
Submit Job with all optional input arguments
Example of AWS Batch Universal Task for reading job status through HTTP
proxy connection.
...
submitting a new AWS Batch with the following arguments.
Environment Variables as AWS Region
Example of
...
Universal Task for
...
HTTPS with password Proxy Type Task Configuration
Example of AWS Batch Universal Task for reading job status through HTTPS with password
proxy connection.
Task Output
Output Only Fields
...
submitting a new AWS Batch Job providing no AWS Credentials in task definition and providing AWS Region as Environment Variable, leaving the respective input fields empty. AWS Credentials are expected in this case to be configured on the task execution environment. Please refer to AWS Credentials input field for more information.
Job Timeout
Container Overrides Script
Wait for Success or Failure
Proxy Type "HTTPS With Credentials"
Read Job Status with Role Based Access Enabled
Example of AWS Batch Universal Task for reading the status of an existing Batch Job by ID, Role Based Access and "HTTP Proxy" connection.
Task Output
Output Only Fields
The output fields for this Universal Extension are described below.
Field | Type | Preserved on re-run | Description |
---|---|---|---|
Job ARN | Text | False | ARN value of the newly submitted Batch Job. Generated in Submit Job action. |
Job ID | Text | False | Id of the newly submitted Batch Job. Generated in Submit Job action. |
Job Name | Text | False | Name of the Batch Job. Since 1.2.0: Generated for Action "Submit Job" with Wait for Success or Failure = "True", updating live during execution. |
Job Status | Text | False | Status of the Batch Job. |
Exit Codes
...
Since 1.2.0: Generated for Action "Submit Job" with Wait for Success or Failure = "True", updating live during execution. |
Exit Codes
The exit codes for AWS Batch Extension are described below.
Exit Code | Status Classification Code | Status Classification Description | Status Description | ||
---|---|---|---|---|---|
0 | SUCCESS | Successful Execution | SUCCESS: Successful Task execution | ||
1 | FAIL | Failed Execution | FAIL: < Error Description > | ||
2 | AUTHENTICATION_ERROR | Bad credentials | AUTHENTICATION_ERROR: Account cannot be authenticated. | ||
3 | AUTHORIZATION_ERROR | Insufficient Permissions | AUTHORIZATION_ERROR: Account is not authorized to perform the requested action. | ||
10 | CONNECTION_ERROR | Bad connection data or connection timed out | CONNECTION_ERROR: < Error Description > | ||
11 | CONNECTION+_ERROR | Extension specific connection error | CONNECTION_ERROR: ProxyConnectionError: Failed to connect to proxy URL <url> | ||
2122 | SUCCESS | Successful Execution | SUCCESS: Job is in FAILEDRUNNING state. 22Valid only for Action = "Read Job" | ||
23 | SUCCESS | Successful Execution | SUCCESS: Job is in RUNNINGSTARTING state. | 23 | Valid only for Action = "Read Job" |
24 | SUCCESS | Successful Execution | SUCCESS: Job is in STARTINGRUNNABLE state. 24Valid only for Action = "Read Job" | ||
25 | SUCCESS | Successful Execution | SUCCESS: Job is in | RUNNABLE PENDING state. | 25Valid only for Action = "Read Job" |
26 | SUCCESS | Successful Execution | SUCCESS: Job is in SUBMITTED state. 25Valid only for Action = "Read Job" | ||
27 | SUCCESS | Successful Execution | SUCCESS: Job is in SUCCEEDED state. |
Extension Output
The Extension Output for AWS Batch Universal Task successful execution is described below.
...
Valid only for Action = "Read Job" |
Attribute changed
is populated as follows
- true, in case Action "Submit Job" is executed successfully.
- false, in case Action "Read Job" is executed.
The Extension output contains attribute result
. Attribute result
contains the following sub-attributes:
Attribute | Type | Description |
---|---|---|
out_job_arn | string | ARN value of the newly submitted Batch Job. Generated in "Submit Job" Action. |
out_job_id | string | ID of the newly submitted Batch Job. Generated in "Submit Job" Action. |
out_job_name | string | Name of the Batch Job. Generated in "Read Job" Action Since 1.2.0: Generated for Action "Submit Job" with Wait for Success or Failure = "True", updating live during execution. |
out_job_status | string | Status of the Batch Job. Generated in "Read Job" Action Since 1.2.0: Generated for Action "Submit Job" with Wait for Success or Failure = "True", updating live during execution. |
job_info | JSON | Latest job information provided from Batch service Generated in "Read Job" and "Submit Job" Action |
Extension Output
The Extension Output for AWS Batch Universal Task successful execution is described below.
{
"exit_code":0,
"status_description":"SUCCESS: AWS Batch Task submitted successfully",
"changed":true,
"invocation":{
"extension":"ue-aws-batch",
"version":"1.3.0",
"fields":{
"action":"Submit Job",
"credentials_user":"****",
"credentials_password":"****",
...
Document References
This document references the following documents:
...
"region":"us-east-1",
"role_based_access":false,
"role_arn":null,
"job_name":"ue-aws-batch-demo-job",
"job_definition":"ue-aws-batch-job-definition",
"job_queue":"ue-aws-batch-test-queue",
"job_timeout":100,
"additional_job_parameters":[
],
"job_id":null,
"use_proxy":false,
"proxy_type":null,
"proxy":null,
"proxy_credentials_user":null,
"proxy_credentials_password":null,
"proxy_ca_bundle_file":null,
"container_overrides_script":{
"environment":[
{
"name":"ENV_VAR1",
"value":"value 1"
},
{
"name":"ENV_VAR2",
"value":"value 2"
}
]
},
"wait_for_success_or_failure":true,
"polling_interval":3,
"full_job_info": true
}
},
"result":{
"out_job_name":"ue-aws-batch-demo-job",
"out_job_status":"SUCCEEDED",
"out_job_arn":"arn:aws:batch:us-east-1:123456789:job/1234-5678-90123-456-7890",
"out_job_id":"1111-11111-11111-11111-11111",
"job_info: [ ... ]
}
}
Cancelation and Re-Run
There will be no specific cancel logic. In case of Retry, the Batch job will be re-submitted.
STDOUT and STDERR
STDOUT and STDERR provide additional information to User. The populated content can be changed in future versions of this extension without notice. Backward compatibility is not guaranteed.
Document References
This document references the following documents.
Document Link | Description |
---|---|
Universal Templates | User documentation for creating, working with and understanding Universal Templates and Integrations. |
Universal Tasks | User documentation for creating Universal Tasks in the Universal Controller user interface. |
AWS Batch | User guide for AWS Batch. |
IAM RBAC authorization model | User Documentation for Comparing ABAC to the traditional RBAC model. |
Anchor | ||||
---|---|---|---|---|
|
Changelog
ue-aws-batch-1.3.1 (2024-02-01)
Fixes
Fixed:
Revert `certifi` as requirement. (#35294)
ue-aws-batch-1.3.0 (2023-12-11)
Enhancements
Added
: Extension Output is enhanced to provide the latest job information coming from the AWS Batch Service. (#33753)
ue-aws-batch-1.2.1 (2023-08-04)
Fixes
Fixed:
Field 'Proxy Type' raised a data validation error on Controller 7.3.0.0 and later. (#34038)
ue-aws-batch-1.2.0 (2022-07-19)
Enhancements
Added
: Support Submit Batch Job and Wait until Job Reaches status "Succeeded" or "Failed (#29278)Added
: Output fields are updated during execution for the above scenario.Added
: Log payload response for Read and Submit Job Action on debug mode.
ue-aws-batch-1.1.0 (2022-06-09)
Enhancements
Added
: Provide the capabilty to rely on AWS credentials set-up on the environment where the extension is running and therefore it is not mandatory to be passed on the task definition as input fields. The same applies to AWS Region. (#28286)Added
: AWS Region can be resolved from environment variables. (#28286)Added
: Support for "containerOverrides" parameter. (#28860)
ue-aws-batch-1.0.1 (2022-03-14)
Enhancements
Added
: Update of extension icon. (#27753)
Fixes
Fixed
Fix wrong exit code when invalid ARN is provided. (#27736)Fixed
Fix response message when status is SUCCESS. (#27730)