...

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at https://www.stonebranch.com/integration-hub/Terms-and-Privacy/Terms-of-Use/

Introduction

This Universal Task allows customers to create an EC2 instance with parameters, either in task form, or by simply creating an EC2 instance from the existing AWS launch template. This task also offers the option to additionally install a Linux/UNIX Universal Agent in the newly provisioned EC2 Instance.

Overview

• The task interacts with the AWS platform via a Python boto3 module.

• All AWS credentials remain encrypted.

• Customers can also install/configure a Linux Universal Agent for each EC2 instance, enabling the Universal Controller to instantly communicate with the newly created instance. (NOTE: only Linux Universal Agent is supported at the moment.)

• This task also lets customers create multiple EC2 instances with the same configuration. New instances can also be tagged.

• It allows customers to create a new keypair or use an existing one for the new EC2 instance.

• This task also enables options for additional EBS volume and encryption, as well as detailed monitoring. 

AWS EC2 Task High-Level Overview

Image Removed

Overview

AWS Batch is a set of batch management capabilities that enables developers, scientists, and engineers to easily and efficiently run hundreds of thousands of batch computing jobs on AWS.

This Universal Extension provides the capability to submit new AWS Batch Jobs as well as read the status for an existing AWS Batch Job.

Version Information

Refer to ChangeLog for version history information.

Software Requirements

This integration requires an a Universal Agent and a Python runtime to execute the Universal Task against AWS EC2 Instance.

Software Requirements for /wiki/spaces/UC71x/pages/5178050 and /wiki/spaces/UC71x/pages/5180675 Universal Template and Universal Task

Requires Python

...

3.7

...

.

...

0 or higher. Tested with the Universal Agent bundled Python distribution.

...

• requests

• Boto3

Software Requirements for Universal Agent

Both Windows and Linux agents are supported:

• Universal Agent for Windows x64 Version

...

• 7.

...

• 2.0.0 and later with python options installed.
• Universal Agent for Linux Version

...

• 7.

...

• 2.0.0 and later with python options installed.

Software Requirements for Universal Controller

Universal Controller Version

...

7.

...

2.0.0 and later.

Software Requirements for the Application to be Scheduled

• The Server Running the Universal Agent needs to have Python 2.7.x or 3.6.x installed

• AWS IAM Credentials -Access Key, Secret Access key and Region with EC2 set of permissions

• This universal task for the AWS EC2-start-stop-terminate has been tested with the agent bundled with python 3.6 and boto3 module

Technical Considerations

• Consider using this universal task either with universal agent bundled with python(uapy) and also having boto3 module within this environment or a python environment (py) in a host where universal agent is installed with boto3 module in it.

• AWS IAM credentials (Access Key, Secret Access key and Region) should be with the Appropriate access for handling AWS EC2 instances.

• With the current version of this Universal Task, Universal Agent can be installed only in Linux EC2 Instance.

AWS Create EC2 Instance (with Universal Agent) Key Features

...

Feature

...

Description

...

Create New EC2 Instance

...

Creates a EC2 Instance based on the parameters that are provided in the form

...

Launch EC2 from template

...

Create a EC2 Instance based on a template in AWS

Import AWS Create EC2 Instance (with Universal Agent) Downloadable Universal Template

...

Network and Connectivity Requirements

Extensions' Universal Agent host should be able to reach AWS Batch REST endpoints, through the configured VPC.

More information on setting up AWS Batch VPC, can be found in the AWS Batch official user guide.

Key Features

This Universal Extension provides the following key features:

• Support to submit a new Batch Job, with option to Terminate Job after a timeout period.
• Support to submit a new Batch Job and wait until it reaches state "success" or "failed".
• Support to read Batch Job status for an existing Job ID.
• Support for authorization via IAM Role-Based Access Control (RBAC) strategy.
• Support for Proxy communication via HTTP/HTTPS protocol.

Import Universal Template

To use the Universal Template, you first must perform the following steps:

1. This Universal Task requires

...

1. the Resolvable Credentials feature. Check that the

...

1. Resolvable Credentials Permitted system property has been set to true.

...

1. To import the Universal Template

...

In the Universal Controller UI, select Configuration > Universal Templates to display the current list of /wiki/spaces/UC71x/pages/5178054.

...

Right-click any column header on the list to display an Action menu.

...

Select Import from the menu, enter the directory containing the Universal Template file(s) that you want to import, and click OK.

1. into your Controller, follow the instructions here.

2. When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list.

Configure

...

Universal Task

For the a new Universal Task type, create a new task, and enter the task-specific details that were created in the Universal Template.

Field Descriptions for AWS Create EC2 Instance (with Universal Agent) Universal Task

...

Field

...

Description

...

AWS-DEFAULT-REGION

...

AWS Region kept as credential

...

AWS-SECRET-ACCESS-KEY

...

AWS Secret Key

...

AWS-ACCESS-KEY-ID

...

AWS Access Key

...

Launch Instance Option

...

Select either launch from template or create a brand new ec2 instance with the parameter supplied in the form

...

LaunchTemplateName

...

Mandatory if launch_instance_option=” Launch from template”

...

AWS_IMAGE_ID

...

Provide the AWS machine ID ,Mandatory if launch_instance_option=” new_instance”

...

Keypair option

...

PEM file creation choice , Select either existing Key pair or New Key pair

...

EC2-KEYPAIR-Path & Name

...

Provide Keypair file name and the path (Do not give the extension) for new and for existing keypair just the name

...

EC2 Instance Type

...

provide ec2-instance type like t2. Micro , if Launch from template = “Create New Instance”

...

Minimum Count

...

Minimum Count of instance that need to be created, if Launch from template = “Create New Instance”

...

Max Count

...

Max count of instance that needs to be created, if Launch from template = “Create New Instance”

...

associate_public_ip

...

If a public IP needs to be created when a instance is created

...

SubnetId

...

Provide subnetID where the instance to be associated within AWS

...

Availability Zone

...

Provide Availability Zone where the instance to be associated within AWS

...

Security Group ID

...

provide security group ids, if multiple ID's then separate by comma

...

Instance Tag name

...

EC2 Instance Tag Name

...

iam_instance_profile_name

...

If applicable provide the IAM Instance Profile Name

...

device_name

...

Provide the device name; for example, /dev/sda1

...

ebs_volume_size

...

Provide EBS Volume size

...

EBS Vol. Type

...

Select either standard or io1 or gp2 or sc1 or st1

...

EBS Vol. Encyption

...

Check if encryption needs needed

...

EC2 Monitoring

...

Check this box if detailed monitoring required

...

Install Universal Agent

...

Check this box if you would need to install universal agent with this new EC2 instance created

...

Agent Download URL

...

Provide the path to download the agent URL, if install universal agent option is selected

...

Universal Agent Install OS

...

select the OS where universal agent needs to be installed

...

Agent OMS IP

...

Provide the OMS server IP for the universal agent to be connected after installation , if install universal agent option is selected

...

Use Public IP for SSH

...

Select if you would need to use the public or provide IP for SSH

...

os_user_id

...

Provide the OS user ID that will be used to make SSH connection

Examples for AWS Create EC2 Instance (with Universal Agent) Universal Tasks

New EC2 Instance Creation

Image Removed

Launch Instance with Launch Template

Image Removedrequired input fields.

Input Fields

The input fields for this Universal Extension are described below.

Task Examples

Submit Job

Example of AWS Batch Universal Task for submitting a new AWS Batch Job.

Image Added

Submit Job with all optional input arguments

Example of AWS Batch Universal Task for submitting a new AWS Batch with the following arguments.

• Environment Variables as AWS Region

  Example of Universal Task for submitting a new AWS Batch Job providing no AWS Credentials in task definition and providing AWS Region as Environment Variable, leaving the respective input fields empty. AWS Credentials are expected in this case to be configured on the task execution environment. Please refer to AWS Credentials input field for more information.

• Job Timeout

• Container Overrides Script

• Wait for Success or Failure

• Proxy Type "HTTPS With Credentials"

Image Added

Read Job Status with Role Based Access Enabled

Example of AWS Batch Universal Task for reading the status of an existing Batch Job by ID, Role Based Access and "HTTP Proxy" connection.

Image Added

Task Output

Output Only Fields

The output fields for this Universal Extension are described below.

Exit Codes

The exit codes for AWS Batch Extension are described below.

Attribute changed is populated as follows

• true, in case Action "Submit Job" is executed successfully.
• false, in case Action "Read Job" is executed.

The Extension output contains attribute result. Attribute result contains the following sub-attributes:

Extension Output

The Extension Output for AWS Batch Universal Task successful execution is described below.

{
   "exit_code":0,
   "status_description":"SUCCESS: AWS Batch Task submitted successfully",
   "changed":true,
   "invocation":{
      "extension":"ue-aws-batch",
      "version":"1.3.0",
      "fields":{
         "action":"Submit Job",
         "credentials_user":"****",
         "credentials_password":"****",
         "region":"us-east-1",
         "role_based_access":false,
         "role_arn":null,
         "job_name":"ue-aws-batch-demo-job",
         "job_definition":"ue-aws-batch-job-definition",
         "job_queue":"ue-aws-batch-test-queue",
         "job_timeout":100,
         "additional_job_parameters":[
            
         ],
         "job_id":null,
         "use_proxy":false,
         "proxy_type":null,
         "proxy":null,
         "proxy_credentials_user":null,
         "proxy_credentials_password":null,
         "proxy_ca_bundle_file":null,
         "container_overrides_script":{
            "environment":[
               {
                  "name":"ENV_VAR1",
                  "value":"value 1"
               },
               {
                  "name":"ENV_VAR2",
                  "value":"value 2"
               }
            ]
         },
         "wait_for_success_or_failure":true,
         "polling_interval":3,
         "full_job_info": true

      }
   },
   "result":{
      "out_job_name":"ue-aws-batch-demo-job",
      "out_job_status":"SUCCEEDED",
      "out_job_arn":"arn:aws:batch:us-east-1:123456789:job/1234-5678-90123-456-7890",
      "out_job_id":"1111-11111-11111-11111-11111",
      "job_info: [ ... ]
   }
}

Cancelation and Re-Run

There will be no specific cancel logic. In case of Retry, the Batch job will be re-submitted.

STDOUT and STDERR

STDOUT and STDERR provide additional information to User. The populated content can be changed in future versions of this extension without notice. Backward compatibility is not guaranteed.

Document References

This document references the following documents:.

Changelog

ue-aws-batch-1.3.1 (2024-02-01)

Fixes

• Fixed: Revert `certifi` as requirement. (#35294)

ue-aws-batch-1.3.

...

Universal Tasks

...

https://docs.stonebranch.com/confluence/display/UC71x/Universal+Tasks

...

0 (2023-12-11)

Enhancements

• Added: Extension Output is enhanced to provide the latest job information coming from the AWS Batch Service. (#33753)

ue-aws-batch-1.2.1 (2023-08-04)

Fixes

• Fixed: Field 'Proxy Type' raised a data validation error on Controller 7.3.0.0 and later. (#34038)

ue-aws-batch-1.2.0 (2022-07-19)

Enhancements

• Added: Support Submit Batch Job and Wait until Job Reaches status "Succeeded" or "Failed (#29278)
• Added: Output fields are updated during execution for the above scenario.
• Added: Log payload response for Read and Submit Job Action on debug mode.

ue-aws-batch-1.1.0 (2022-06-09)

Enhancements

• Added: Provide the capabilty to rely on AWS credentials set-up on the environment where the extension is running and therefore it is not mandatory to be passed on the task definition as input fields. The same applies to AWS Region. (#28286)
• Added: AWS Region can be resolved from environment variables. (#28286)
• Added: Support for "containerOverrides" parameter. (#28860)

ue-aws-batch-1.0.1 (2022-03-14)

Enhancements

• Added: Update of extension icon. (#27753)

Fixes

• Fixed Fix wrong exit code when invalid ARN is provided. (#27736)
• Fixed Fix response message when status is SUCCESS. (#27730)