Disclaimer

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at https://www.stonebranch.com/integration-hub/Terms-and-Privacy/Terms-of-Use/

Overview

AWS Glue is a serverless data-preparation service for extract, transform, and load (ETL) operations. It makes it easy for data engineers, data analysts, data scientists, and ETL developers to extract, clean, enrich, normalize, and load data.

This Universal Extension provides the capability to submit a new AWS Glue Job.

Version Information

Refer to Changelog for version history information.

This integration requires a Universal Agent and a Python runtime to execute the Universal Task.

Software Requirements for Universal Template and Universal Task

Tested with Python version 3.7.6 and 3.11.6 and with the Universal Agent bundled Python distribution.

Software Requirements for Universal Agent

Both Windows and Linux agents are supported.

• Universal Agent for Windows x64 Version >= 7.4.0.0 
• Universal Agent for Linux Version >= 7.4.0.0 

Software Requirements for Universal Controller

Universal Controller Version >=7.4.0.0

Network and Connectivity Requirements

Extension's Universal Agent host should be able to reach AWS Glue REST endpoints. The AWS Credentials provided in the AWS Glue Universal Task, should have sufficient permissions on AWS to invoke Glue Jobs.

Key Features

This Universal Extension provides the following key features.

• Actions
  • Start a Glue job.
  • Start a Glue job and wait until it reaches state "success" or "failed".
• Authentication
  • Authentication through HTTPS
  • Authentication through IAM Role-Based Access Control (RBAC) strategy.
• Input/Output
  • Option to pass Input Arguments as UAC script supporting UAC environment variables and UAC Functions.
• Other
  • Support for Proxy communication via HTTP/HTTPS protocol.

Import Universal Template

To use the Universal Template, you first must perform the following steps.

1. This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.

2. To import the Universal Template into your Controller, follow the instructions here.

3. When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list.

Modifications of this integration, applied by users or customers, before or after import, might affect the supportability of this integration. For more information refer to Integration Modifications.

Configure Universal Task

For a new Universal Task, create a new task, and enter the required input fields.

Input Fields

The input fields for this Universal Extension are described in the following table.

Task Examples

Start Job Run

Start a new job run.

Start Job Run with all optional input arguments

Start a new Job Run for a given Run ID (retries a previous execution), with all optional input argument.

Start Job Run with all optional input arguments and script

Start a new Job Run for a given Run ID (retries a previous execution), with all optional input argument as above but use "Script" as Input Arguments Source.

Job arguments in UAC Script in JSON format can pass arguments from UAC Variables or UAC Functions as shown below. More information about escaping characters for json format here.

Start Job Run with Endpoint URL

Start a new Job Run, overriding the default AWS Endpoint.

Start Job Run with Role ARN and Proxy configuration

Start a new Job Run assuming a provided ARN Role, and also using a Proxy configuration.

Start Job Run with Environment Variables as Region

Start a new job run, providing no AWS Credentials in task definition and providing AWS Region as Environment Variable, leaving the respective input fields empty. AWS Credentials are expected in this case to be configured on the task execution environment. Please refer to AWS Credentials input field for more information.

Task Output

Output Only Fields

The output fields for this Universal Extension are described below.

Exit Codes

The exit codes for the Extension are described below.

Extension Output

In the context of a workflow, subsequent tasks can rely on the information provided by this integration as Extension Output.

Attribute changed is populated as follows.

• true in case the job is triggered successfully
• false otherwise

result section includes the following attributes.

An example of the Extension Output with Wait for Success or Failure = "False" for a successful triggering job is presented below.

{
    "exit_code": 0,
    "status_description": "SUCCESS: AWS Glue Job started successfully.",
    "changed": true,
    "invocation": {
        "extension": "ue-aws-glue",
        "version": "2.1.0",
        "fields": { ... }
    },
    "result": {
        "out_job_run_id": "jr_123456789"
    }
}

An example of the Extension Output with Wait for Success or Failure = "True" for a successful triggering job is presented below.

{
    "exit_code": 0,
    "status_description": "SUCCESS: AWS Glue Job started successfully and resulted in status SUCCEEDED.",
    "changed": true,
    "invocation": {
        "extension": "ue-aws-glue",
        "version": "2.1.0",
        "fields": { ... }
    },
    "result": {
        "job_run_id": "jr_57133f7bb82f13a29fa8813d95e2b941a3c6f5f67475227e1bb8d213e888478c",
        "job_run_status": "SUCCEEDED",
        "started_on": "2024-03-27 15:26:30.998000+02:00",
        "last_modified_on": "2024-03-27 15:26:58.791000+02:00",
        "completed_on": "2024-03-27 15:26:58.791000+02:00",
        "error_message": null
    }
}

STDOUT and STDERR

STDOUT and STDERR provide additional information to the user.

Extensions Cancellation and Re-Run

• Canceling a task in UAC will only cancel it in UAC and will not have any effect on the running AWS Glue Job.
• Re-Running a task in UAC will execute the task again and start a new AWS Glue Job.

Integration Modifications

Modifications applied by users or customers, before or after import, might affect the supportability of this integration. The following modifications are discouraged to retain the support level as applied for this integration.

• Python code modifications should not be done.
• Template Modifications
  • General Section
    • "Name", "Extension", "Variable Prefix", "Icon" should not be changed.
  • Universal Template Details Section
    • "Template Type", "Agent Type", "Send Extension Variables", "Always Cancel on Force Finish" should not be changed.
  • Result Processing Defaults Section
    • Success and Failure Exit codes should not be changed.
    • Success and Failure Output processing should not be changed.
  • Fields Restriction Section
    The setup of the template does not impose any restrictions, However with respect to "Exit Code Processing Fields" section.
    1. Success/Failure exit codes need to be respected.
    2. In principle, as STDERR and STDOUT outputs can change in follow-up releases of this integration, they should not be considered as a reliable source for determining success or failure of a task.

Users and customers are encouraged to report defects, or feature requests at Stonebranch Support Desk.

Document References

This document references the following documents:

Changelog

ue-aws-glue-2.1.0 (2024-08-29)

Enhancements

• Added: new input field - Endpoint URL (#41648, #118162)

Fixes

• Fixed: "Proxy Type" field incorrectly used. It is not required to be filled anymore by users on task definition and from this version onwards it is hidden and not used (#41745)

ue-aws-glue-2.0.0 (2024-04-18) 

Deprecations and Breaking Changes

• Breaking Change: drop support for agent 7.3.X or lower, agent version 7.4.X or higher is required

Enhancements

• Added: new input field - Execution Class
• Added: newer worker types compatible with AWS Glue version 3, are now supported 

ue-aws-glue-1.2.1 (2023-12-21)

Fixes

• Fixed: auto-renew AWS temporary credentials before expiration when using ARN based access
• Fixed: fixed issue with polling logic where task would get stuck with status Running (#35135)

ue-aws-glue-1.2.0 (2022-11-11)

Enhancements

• Added: Support Start Glue Job and Wait until Job Reaches status "Succeeded" or "Failed" (#30157)
• Added: Larger set of output fields (#30157)
• Added: Log payload response for Job Run Status and Start Glue Job Run Action on debug mode.
• Added: Option to pass Input Arguments as UAC script supporting UAC environment variables and UAC Functions.

ue-aws-glue-1.1.0 (2022-06-23)

Enhancements

• Added: Allow AWS Credentials and AWS Region as optional fields enabling their configuration on the task execution environment. (#28312)

ue-aws-glue-1.0.0 (2022-03-31)

Initial Version