Disclaimer

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at https://www.stonebranch.com/integration-hub/Terms-and-Privacy/Terms-of-Use/

Overview

AWS Glue is a serverless data-preparation service for extract, transform, and load (ETL) operations. It makes it easy for data engineers, data analysts, data scientists, and ETL developers to extract, clean, enrich, normalize, and load data.

This Universal Extension provides the capability to submit a new AWS Glue Job.

Software Requirements

This integration requires a Universal Agent and a Python runtime to execute the Universal Task.

Software Requirements for Universal Template and Universal Task

Requires Python 3.7.0 or higher. Tested with the Universal Agent bundled Python distribution.

Software Requirements for Universal Agent

Both Windows and Linux agents are supported:

• Universal Agent for Windows x64 Version 7.0.0.0 and later with python options installed.
• Universal Agent for Linux Version 7.0.0.0 and later with python options installed.

Software Requirements for Universal Controller

Universal Controller Version 7.0.0.0 and later.

Network and Connectivity Requirements

Extension's Universal Agent host should be able to reach AWS Glue REST endpoints. The AWS Credentials provided in the AWS Glue Universal Task, should have sufficient permissions on AWS to invoke Glue Jobs.

Key Features

This Universal Extension provides the following key features:

• Start a Glue job.
• Support authorization via IAM Role-Based Access Control (RBAC) strategy.
• Support Proxy communication via HTTP/HTTPS protocol.

Import Universal Template

To use the Universal Template, you first must perform the following steps:

1. This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.

2. Download the provided ZIP file.

3. In the Universal Controller UI, select Administration >Configuration > Universal Templates to display the current list of Universal Templates.

4. Click Import Template.

5. Select the template ZIP file and Import.

When the template has been imported successfully, the Universal Template will appear on the list, refresh your Navigation Tree to see these tasks in the Automation Center Menu.

Configure Universal Task

For the new Universal Task type AWS Glue, create a new task, and enter the task-specific details that were created in the Universal Template.

Input Fields

The input fields for this Universal Extension are described in the following table.

Extension Cancellation

When using a 7.0 or newer template, we must ensure that the “Always Cancel On Force Finish” is checked. This is to minimize leaving “orphan” processes on the OS without the option for the agent to see they are running.

Task Examples

Start Job Run with only required arguments

Start a new Glue job run, providing the only required field Job Name.

Start Job Run with all optional input arguments

Start a new Glue Job Run for a given Run ID (retries a previous execution), with all optional input argument.

Start Job Run with Role ARN and Proxy configuration

Start a new Glue Job Run assuming a provided ARN Role, and also using a Proxy configuration.

Exit Codes

The exit codes for AWS Lambda Extension are described below.

Extension Output

The Extension Output for AWS Lambda Universal Task successful execution is described below.

{
    "exit_code": 0,
    "status_description": "SUCCESS: AWS Glue Job started successfully",
    "changed": true,
    "invocation": {
        "extension": "ue-aws-glue",
        "version": "1.0.0",
        "fields": {
            "action": "Start Job Run",
            "aws_credentials_user": "test-user",
            "aws_credentials_password": "****",
            "region": "us-east-1",
            "role_based_access": false,
            "role_arn": null,
            "job_name": "TestJob1",
            "job_run_id": null,
            "security_config": null,
            "worker_type": "G.1X",
            "num_workers": 3,
            "job_timeout": 2880,
            "notify_delay_period": 3,
            "input_arguments": [
                {
                    "Stonebranch": "Extension"
                }
            ],
            "use_proxy": true,
            "proxy": "https://proxy.example.com:8080",
            "proxy_type": "HTTPS",
            "proxy_ca_bundle_file": "/tmp/proxy_ca.pem",
            "proxy_credentials_user": null,
            "proxy_credentials_password": null
        }
    },
    "result": {
        "out_job_run_id": "jr_c83819e1ded81e44fc05d8bfbbf9394b9c9edc7693312d0be05d51ab2fd921c7"
    }
}

Document References

This document references the following documents: