Disclaimer

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at https://www.stonebranch.com/integration-hub/Terms-and-Privacy/Terms-of-Use/

Overview

AWS Glue is a serverless data-preparation service for extract, transform, and load (ETL) operations. It makes it easy for data engineers, data analysts, data scientists, and ETL developers to extract, clean, enrich, normalize, and load data.

This Universal Extension provides tha capability to submit a new AWS Glue Job.

Software Requirements

This integration requires a Universal Agent and a Python runtime to execute the Universal Task.

Software Requirements for Universal Template and Universal Task

Requires Python 3.7.0 or higher. Tested with the Universal Agent bundled Python distribution.

Software Requirements for Universal Agent

Both Windows and Linux agents are supported:

• Universal Agent for Windows x64 Version 7.1.0.0 and later.
• Universal Agent for Linux Version 7.1.0.0 and later.

Software Requirements for Universal Controller

Universal Controller Version 7.1.0.0 and later.

Key Features

This Universal Extension provides the following key features:

• Support for communicating with APIs using the HTTP(S)/REST protocol.
• Support for Authorization towards the foreign API using Basic Authentication or OAuth2 token.
• Support for HTTP Methods GET, POST, PUT, DELETE, PATCH.
• Support for SSL Protocol.
• Ability to use proxy between Universal Controller and target web service.
• Ability for configuration of custom exit codes based on the web service response payload (Supported for JSON payloads).

Import Universal Template

To use the Universal Template, you first must perform the following steps:

1. This Universal Task requires the Resolvable Credentials feature, check that the Resolvable Credentials Permitted system property has been set to true. For more information about Resolvable Credentials click here.

2. Download the provided ZIP file.

3. In the Universal Controller UI, select Administration >Configuration > Universal Templates to display the current list of Universal Templates.

4. Click Import Template.

5. Select the template ZIP file and Import.

When the template has been imported successfully, the Universal Template will appear on the list, refresh your Navigation Tree to see these tasks in the Automation Center Menu.

Configure Universal Task

For the new Universal Task type, create a new task, and enter the task-specific details that were created in the Universal Template.

Input Fields

The input fields for this Universal Extension are described in the following table.

SSL Certificate Authority File Discovery

• If Python 3.7 Distribution for Universal Agent is installed, the path location of the Certificate Authority file must be provided though Trusted Certificates File input field.
• If Python is installed on the local system but not with the Universal Agent Distribution:
  • If the SSL certificate is signed by a publicly trusted certificate authority (CA), it is not required to provide the certificate.
  • If the SSL certificate is self-signed, either the path location of the certificate file must be provided, or the certificate must be stored at the machine's default location for installed certificates (in this case it will be trusted for all applications).
• As an alternative to the population of Trusted Certificates File input field, environment variable "REQUESTS_CA_BUNDLE" can be used, which points to the Trusted Certificates File path.

Exit Code Mapping

As part of task configuration, it is possible for the user to define custom exit codes that depend on the contents of web service response body. This might be useful in the following cases:

• When the task is used in a workflow and, based on the contents of the web service response body, the workflow needs to follow a different path. Exit codes can be used in workflows to follow different branches.

* When retry needs to be configured, based on the content of the web service response body. Task retry can be configured based on exit code.
For example, a user can configure the Universal Controller task to retry execution when Exit code is X. Exit code X can be configured by the user when web service response payload field JSON "document.status" is equal to value "Under Processing".

The fields related to this functionality are the following:

• Process Exit Code Mapping
  Should be checked to enable this functionality.
• Path Expression
  The JSON Path ( JSONPath Syntax).
• Response Exit Code Mapping
  A list of pairs (Pattern/Exit Code). If pattern is matched, then the configured Exit Code will be applied.

The extension will parse the webservice output JSON body and get the value returned by the Path Expression This value will be compared against the list of patterns provided on Response Exit code Mapping array. When a match is found, the extension will return the corresponding Exit Code related to the matched pattern.

The following features and restrictions are applied:

• It is possible to provide multiple pairs of Pattern/Exit Code. The first match will be considered.
• It is possible to provide an asterisk (*) as a pattern. This will tell the extension to match any value found (catch all).
• Provided exit codes should be between 100 and 255.
• An error will be raised in case:
  • No match found.
  • Wrong Path Expression is provided.
  • The result body is not of type JSON.
• The custom exit codes that from a business perspective are resulting in the success of the task, need to be configured during task configuration or the task will be marked as failed. Configuration can be found under the field group Result Processing Details:
  • Exit Code Processing
    Option "Success Exit Code Range" to be selected in this field.
  • Exit Codes
    To be filled with the list of exit codes that the user expects to result success task (Ranges can be provided. For example: 2,100-255).

Task Examples

No Authorization Task Configuration

Example of Universal Task for "None" Authorization Type.

Basic Authorization Task Configuration

Example of Universal Task for "Basic" Authorization Type.

OAuth2 Token Authorization Task Configuration

Example of Universal Task for "OAuth2 Token" Authorization Type. A token can be fetched with the OAuth2 Token Universal Extension, saved as a variable and used from this Universal Extension.

Task Configuration for SSL Protocol

Example of Universal Task for Use SSL checked.

Task Configuration for POST request with JSON payload and use of proxy server

Example of Universal Task for Use SSL checked. JSON payload used:

{
  "first name": "John",
  "last name": "Smith"
}

Printing Result Body to STDOUT

Example showing the result body of the webservice printed to STDOUT.

Response Code Output Field

Example showing the value of the output field "Response Code" filled with the actual response code from webservice result.

Task configuration for exit code mapping scenario

The following example scenario describes the configuration of a task where the user expects an exit code equal to "101" if the first contract (contracts[0]) of the "contracts" list has a firstName = "Tom".
Task inputs for Web Service Integration Details.

Task inputs for Result Processing Details:
Here the Exit Codes field is filled with two elements, "0" and the range [100-255] where the expected exit code ("101") falls in.

Task Results:

Task Output

Output Fields

Exit Codes

The exit codes for this Universal Extension are described in the following table.

Extension Output

The extension output result has the following fields:

• code
  Response's exit code.
• headers
  List of the response's HTTP headers.
• body
  Body of the response.

An example of Extension Output for this Universal Extension is below:

{
    "exit_code": 101,
    "status_description": "SUCCESS: Task executed successfully with mapped exit code.",
    "invocation": {
        "extension": "ue-webservice",
        "version": "1.0.0",
        "fields": {
            "protocol": "HTTP_REST",
            "http_version": "1.1",
            "authorization_type": "None",
            "credentials_user": null,
            "credentials_password": null,
            "oauth2_token": "",
            "add_authorization_data_to": null,
            "url": "http://ue-wiremock-dev-noauth.stonebranch.org:8080/v1/contacts",
            "http_method": "GET",
            "timeout": null,
            "url_query_parameters": {},
            "payload_type": null,
            "payload_source": null,
            "mime_type": null,
            "other_value_for_mime_type": "",
            "form_data": {},
            "payload": "",
            "payload_script": null,
            "http_headers": {},
            "proxies": "",
            "use_ssl": false,
            "trusted_certificates_file": "",
            "ssl_hostname_check": false,
            "private_key_certificate": "",
            "public_key_certificate": "",
            "result_body_medium": "None",
            "path_expression": "contacts.[0].firstName",
            "exit_code_mapping": {
                "Tom": "101"
            },
            "process_exit_code_mapping": true
        }
    },
    "result": {
        "code": 200,
        "headers": [
            {
                "header": "Content-Type",
                "value": "application/json"
            },
            {
                "header": "Matched-Stub-Id",
                "value": "cf796328-38b8-4200-b0e1-c3ac5a83532b"
            },
            {
                "header": "Matched-Stub-Name",
                "value": "GET all contacts"
            },
            {
                "header": "Vary",
                "value": "Accept-Encoding, User-Agent"
            },
            {
                "header": "Content-Encoding",
                "value": "gzip"
            },
            {
                "header": "Transfer-Encoding",
                "value": "chunked"
            }
        ],
        "body": "{\n  \"contacts\": [\n    {\n      \"id\": \"11111\",\n      \"firstName\": \"Tom\",\n      \"lastName\": \"Smith\",\n      \"email\": \"tom.smith@example.com\",\n      \"dateAdded\": \"2021-01-03\",\n      \"companyId\": \"123\"\n    },\n    {\n      \"id\": \"22222\",\n      \"firstName\": \"Suki\",\n      \"lastName\": \"Patel\",\n      \"email\": \"spatel@example.com\",\n      \"dateAdded\": \"2020-11-12\",\n      \"companyId\": \"123\"\n    },\n    {\n      \"id\": \"33333\",\n      \"firstName\": \"Lexine\",\n      \"lastName\": \"Barnfield\",\n      \"email\": \"barnfield8@example.com\",\n      \"dateAdded\": \"2021-01-03\",\n      \"companyId\": \"234\"\n    }\n  ]\n}"
    }
}

Document References

This document references the following documents: