...
This Universal Extension provides the capability to submit a new AWS Glue Job.
Version Information
Template Name | Extension Name | Extension Version |
---|---|---|
AWS Glue | ue-aws-glue | 1.1.0 |
Refer to Changelog for version history information.
Software Requirements
This integration requires a Universal Agent and a Python runtime to execute the Universal Task.
...
Both Windows and Linux agents are supported:.
- Universal Agent for Windows x64 Version 7.0.0.0 and later with python options installed.
- Universal Agent for Linux Version 7.0.0.0 and later with python options installed.
...
Extension's Universal Agent host should be able to reach AWS Glue REST endpoints. The AWS Credentials provided in the AWS Glue Universal Task, should have sufficient permissions on AWS to invoke Glue Jobs.
Key Features
This Universal Extension provides the following key features:
...
To use the Universal Template, you first must perform the following steps:
This Universal Task requires the Resolvable Credentials feature
. Check, check that the Resolvable Credentials Permitted system property has been set to true.
Download the provided ZIP file.
In the Universal Controller UI, select Administration >Configuration > Universal Templates to display the current list of Universal Templates.
Click Import Template.
Select the template ZIP file and Import.
...
For more information about Resolvable Credentials click here.
To import the Universal Template into your Controller, follow the instructions here.
When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list
...
.
Configure Universal Task
For the a new Universal Task type AWS Glue, create a new task, and enter the task-specific details that were created in the Universal Templaterequired input fields.
Input Fields
The input fields for this Universal Extension are described in the following table.
Field | Input type | Default value | Type | Description | |||||
---|---|---|---|---|---|---|---|---|---|
Action | Required | Start Job Run | Choice | Action The action performed upon the task execution.Available action: The available actions are as follows.
| |||||
AWS Region Required Optional since version 1.1.0 | Optional | - | Text | Region for the Amazon Web Service. Find more information about the AWS Service endpoints and quotas here. When AWS Region is not populated as part of the task definition, during task execution the integration will look for credentials on the task execution environment. Refer to configuration options for more information. | |||||
AWS Credentials Required Optional since version 1.1.0 | Optional | - | Credentials | The Credentials definition should be as follows.
| |||||
Role Based Access | Optional | False | Boolean | Special type of authorization is provided by Role Assumption where the client sends his own credentials and the role he wants to assume from another user. If allowed, the client receives temporary credentials with limited time access to some resources. | |||||
Role ARN | Optional | - | Text | Role Arn: Amazon Role, which is applied for the connection. Role ARN format: Example RoleArn: arn:aws:iam::119322085622:role .Required when Role Based Access="True". | |||||
Job Name | Required | - | Text | Name of the Glue job that will be invoked. | |||||
Job Run ID | Optional | - | Text | ID of a previous Job Run to retry. | |||||
Security Configuration | Optional | - | Text | Name of the Security Configuration structure to be used with the Job Run. | |||||
Worker Type | Optional | None | Choice | Type of predefined worker that is allocated when a job runs. Available options are :the following.
| |||||
Number Of Workers | Optional | - | Integer | Number of workers of a defined Worker Type that are allocated when a job is executed. The maximum number of workers that can be defined are :as follows.
| |||||
Job Timeout | Optional | 2880 | Integer | Job Run timeout in minutes.
| |||||
Notify Delay Period | Optional | - | Integer | After a job run starts, the number of minutes to wait before sending a job run delay notification. | |||||
Input Arguments | Optional | - | Array | Job arguments specifically for this run. For this Job Run, they replace the default arguments set in the job definition itself. | |||||
Proxy Type | Optional | HTTP | Choice | Type of proxy connection to be used. Available options are :the following.
| |||||
Proxy | Optional | - | Text | Comma - separated list of Proxy servers. Valid formats :are the following.
| |||||
Proxy CA Bundle File | Optional | - | Text | Path The path to a custom certificate bundle to use when establishing SSL/TLS connections with proxy. Used when Proxy Type is configured for "HTTPS" or "HTTPS With Credentials". | |||||
Proxy Credentials | Optional | - | Credentials | Credentials to be used for the proxy communication. They are comprised of: The credential definition should be as follows.
Required when "Proxy Type" is configured for "HTTPS" or "HTTPS With Credentials". |
Extension Cancellation
...
. |
...
Task Examples
Start Job Run
...
Start a new Glue job run, providing the only required field Job Name.
Start Job Run with all optional input arguments
Start a new Glue Job Run for a given Run ID (retries a previous execution), with all optional input argument.
Start Job Run with Role ARN and Proxy configuration
Start a new Glue Job Run assuming a provided ARN Role, and also using a Proxy configuration..
Start Job Run with Environment Variables as Region
Start a new job run, providing no AWS Credentials in task definition and providing AWS Region as Environment Variable, leaving the respective input fields empty. AWS Credentials are expected in this case to be configured on the task execution environment. Please refer to AWS Credentials input field for more information.
Task Output
Output Only Fields
The output fields for this Universal Extension are described below.
Field | Type | Description |
---|---|---|
Job Run ID | text | ID of the started job run |
Exit Codes
The exit codes for AWS Lambda the Extension are described below.
Exit Code | Status Classification Code | Status Classification Description | Status Description |
---|---|---|---|
0 | SUCCESS | Successful Execution | SUCCESS: Successful Task execution. |
1 | FAIL | Failed Execution | FAIL: < Error Description >. |
2 | AUTHENTICATION_ERROR | Bad credentials | AUTHENTICATION_ERROR: Account cannot be authenticated. |
3 | AUTHORIZATION_ERROR | Insufficient Permissions | AUTHORIZATION_ERROR: Account is not authorized to perform the requested action. |
10 | CONNECTION_ERROR | Bad connection data or connection timed out | CONNECTION_ERROR: < Error Description >. |
11 | CONNECTION_ERROR | Extension specific connection error | CONNECTION_ERROR: ProxyConnectionError: Failed to connect to proxy URL <url> . |
20 | DATA_VALIDATION_ERROR | Input fields validation error | DATA_VALIDATION_ERROR: Some of the input fields cannot be validated. See STDERR for more details. |
Extension Output
In the context of a workflow, subsequent tasks can rely on the information provided by this integration as Extension Output
...
The Extension Output for AWS Lambda Universal Task successful execution is described .
Attribute changed
is populated as follows.
- true in case the job is triggered successfully
- false otherwise
result
section includes the following attributes.
Attribute | Type | Description |
---|---|---|
out_job_run_id | string | ID of the started job run |
An example of the Extension Output for a successful triggering job is presented below.
{
"exit_code": 0,
"status_description": "SUCCESS: AWS Glue Job started successfully",
"changed": true,
"invocation": {
"extension": "ue-aws-glue",
"version": "1.01.0",
"fields": {
"action": "Start Job Run",
"aws_credentials_user": "test-user",
"aws_credentials_password": "****",
"region": "us-east-1",
"role_based_access": false,
"role_arn": null,
"job_name": "TestJob1",
"job_run_id": null,
"security_config": null,
"worker_type": "G.1X",
"num_workers": 3,
"job_timeout": 2880,
"notify_delay_period": 3,
"input_arguments": [
{
"Stonebranch": "Extension"
}
],
"use_proxy": true,
"proxy": "https://proxy.example.com:8080",
"proxy_type": "HTTPS",
"proxy_ca_bundle_file": "/tmp/proxy_ca.pem",
"proxy_credentials_user": null,
"proxy_credentials_password": null
}
},
"result": {
"out_job_run_id": "jr_c83819e1ded81e44fc05d8bfbbf9394b9c9edc7693312d0be05d51ab2fd921c7"
}
}
STDOUT and STDERR
STDOUT and STDERR provide additional information to User. The populated content can be changed in future versions of this extension without notice. Backward compatibility is not guaranteed.
Document References
This document references the following documents:
NameDocument Link | Location | Description | ||
---|---|---|---|---|
Universal Templates | https://docs.stonebranch.com/confluence/display/UC72x/Universal+Templates | User documentation for creating, working with and understanding Universal Templates in the Universal Controller user interface. | Universal Tasks | https://docs.stonebranch.com/confluence/display/UC72x/Universal+and Integrations. |
Universal Tasks | User documentation for creating Universal Tasks in the Universal Controller user interface. | |||
AWS Glue | https://docs.aws.amazon.com/glue/?id=docs_gateway | Documentation for AWS Lambda. | ||
IAM RBAC authorization model | https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction_attribute-based-access-control.html#introduction_attribute-based-access-control_compare-rbac | User Documentation for Comparing ABAC to the traditional RBAC model.Credentials | User documentation for creating and working with credentials. | |
Resolvable Credentials Permitted Property | User documentation for Resolvable Credentials Permitted Property. |
Changelog
ue-aws-glue-1.1.0 (2022-06-23)
Enhancements
Added
: Allow AWS Credentials and AWS Region as optional fields enabling their configuration on the task execution environment. (#28312)
ue-aws-glue-1.0.0 (2022-03-31)
Initial Version