Table of Contents

indent	5

Refer to Changelog for version history information.

Overview

Amazon EMR (Elastic MapReduce) is a cloud-based service offered by Amazon Web Services (AWS) designed to process and analyze large data sets quickly and cost-effectively. Amazon EMR notebooks. AWS EMR Notebooks are Jupyter-based notebooks that are integrated with Amazon EMR, providing execution capabilities and in addition an interactive environment for data exploration and analysis.

This integration provides the ability the start an AWS EMR notebook execution from UAC and optionally monitor it's execution. It also allows Task authors to specify a large variety of notebook execution configuration options.

...

Name

Type

Description

Version Information

Action

Choice

The action performed upon the task execution. Available options:

Start Notebook Execution (default)

Default value is “Start Notebook Execution”. Note that “Notebook” is an entity on AWS that later was renamed to “Workspace”. However in AWS documentation both terms are used.

1.0.0

Region

Text

Region for the Amazon Web Service i.e. "us-east-1"

When AWS Region is not populated as part of the task definition, during task execution the integration will look for AWS Region on the task execution environment (AWS configuration file or through AWS environment variables).

1.0.0

Credentials

The Credentials definition should be as follows.

AWS Access Key ID as "Runtime User".
AWS Secret Access Key as "Runtime Password".

When AWS Credentials is not populated as part of the task definition, during task execution the integration will look for AWS Credentials on the task execution environment. Refer to AWS Credential Configuration Options for more information

1.0.0

Assume Role ARN

Large Text

Role ARN of the AWS Assume Role functionality. The Assume Role functionality in AWS (Amazon Web Services) allows a user or service to take on the permissions of another IAM (Identity and Access Management) role temporarily.

If field is left empty Role Assumption is notperformed.

1.0.0

Service Role

Text

The name or ARN of the IAM role that is used as the service role for Amazon EMR (the Amazon EMR role) for the notebook execution.

1.0.0

Execution Engine ID

Text

The unique identifier of the execution engine. For an EMR cluster, this is the cluster ID.

1.0.0

Editor ID

Text

The unique identifier of the EMR Notebook to use for notebook execution.

1.0.0

Relative Path

Text

The path and file name of the notebook file for this execution, relative to the path specified for the EMR Notebook.

For example, if you specify a path of "s3://MyBucket/MyNotebooks" when you create an EMR Notebook for a Notebook with an ID ofe-ABCDEFGHIJK1234567890ABCD (the Editor ID of mentioned above), and you specify a Relative Path of "my_notebook_executions/notebook_execution.ipynb" , the location of the file for the notebook execution is "s3://MyBucket/MyNotebooks/e-ABCDEFGHIJK1234567890ABCD/my_notebook_executions/notebook_execution.ipynb"

1.0.0

Notebook Parameters (JSON)

Large Text

Input parameters in JSON format passed to the Amazon EMR Notebook at runtime for execution.

Code Block

title	Example Input
linenumbers	true
collapse	true

{
"number": 30,
"word": "random text",
"random_dict": {"random_key": "text"}
}

1.0.0

Wait for Completion

Checkbox

A switch that controls whether the UAC task will wait until the Notebook Execution is completed.

Default value is unchecked.

1.0.0

Polling Interval

Integer

The time (in seconds) between retries for getting the status of Notebook Execution.

Mandatory if Wait for Success or Failure is “true”

As a best practice, if the Notebook Execution expected completion duration is long, set the polling Interval to a larger value. A short value will trigger frequent checks towards AWS which in the case of long-duration jobs is inefficient in terms of resources.

Default value is 15.

1.0.0

Max Number of Polls

Integer

Maximum number of polls. Can be used to control the approximate expected duration of the Notebook Execution (in relation also to Polling Interval (sec))

If left empty the UAC Task Instance will poll indefinitely checking whether the Notebook Execution is completed or resulted in failure.

If the Maximum Number of Polls is reached the exit code of the Universal Task Instance is 40.

Available if Wait for Success or Failure is “true”.

1.0.0

Additional Optional Parameters

Checkbox

A switch that controls if any additional optional parameters and configurations should be passed to the notebook execution.

Default value is unchecked.

1.0.0

Notebook Execution Name

Text

An optional name for the notebook execution.

Available if Additional Optional Parameters is “true”.

1.0.0

Provide Additional Configuration As

Choice

Specifies how additional configuration options for EMR Notebook execution can be defined on the UAC Task.

Available options are:

– None --
As JSON Text (default)
As JSON UAC Script

Available if Additional Optional Parameters is “true”.

1.0.0

Additional Configuration (Text)

Large Text

Additional configuration options for the notebook execution.

Code Block

title	Default Value
linenumbers	true
collapse	true

{
    "NotebookInstanceSecurityGroupId":"string",
    "ExecutionEngine": {
        "MasterInstanceSecurityGroupId": "string",
        "ExecutionRoleArn": "string"
    },
    "Tags":[
        {
            "Key": "string",
            "Value": "string"
        }
    ],
    "NotebookS3Location":{
        "Bucket": "string",
        "Key": "string"
    },
    "OutputNotebookS3Location":{
        "Bucket": "string",
        "Key": "string"
    },
    "EnvironmentVariables":{
        "string": "string"
    }
}

The default value is a placeholder value meant to serve as a way to inform the user on what the available fields are. It is not meant to be used directly as a default value for an execution of the integration.

Task authors can use the default value provided, as a template.

Root JSON elements (for example NotebookInstanceSecurityGroupId) are optional. Consequently if a JSON element is not required, it should be removed.
"string" is a placeholder and needs to be updated with the real value. The value needs to be a JSON string format, which means that depending on the value some characters should be escaped accordingly to JSON grammar.
Information about the functionality of each field can be found on the Official AWS Guide.

Field is available if Provide Additional Configuration As is “As JSON Text”.

1.0.0

Additional Configuration (Script)

Script

Provides the same functionality as Additional Configuration (Text) option with the added benefit of being reusable by saving the configuration as an UAC script.

Available if Provide Additional Configuration As is “As JSON UAC Script”.

1.0.0

...

Note
Backward compatibility is not guaranteed for the content of STDOUT/STDERR and can be changed in future versions without notice

How To

Import Universal Template

To use the Universal Template, you first must perform the following steps.

This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.
To import the Universal Template into your Controller, follow these instructions.
When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list.

Modifications of this integration, applied by users or customers, before or after import, might affect the supportability of this integration. For more information refer to Integration Modifications.

Configure Universal Task

For a new Universal Task, create a new task, and enter the required input fields.

Integration Modifications

Modifications applied by users or customers, before or after import, might affect the supportability of this integration. The following modifications are discouraged to retain the support level as applied for this integration.

Python code modifications should not be done.
Template Modifications
- General Section
  - "Name", "Extension", "Variable Prefix", "Icon" should not be changed.
- Universal Template Details Section
  - "Template Type", "Agent Type", "Send Extension Variables", "Always Cancel on Force Finish" should not be changed.
- Result Processing Defaults Section
  - Success and Failure Exit codes should not be changed.
  - Success and Failure Output processing should not be changed.
- Fields Restriction Section
  The setup of the template does not impose any restrictions, However with respect to "Exit Code Processing Fields" section.
  1. Success/Failure exit codes need to be respected.
  2. In principle, as STDERR and STDOUT outputs can change in follow-up releases of this integration, they should not be considered as a reliable source for determining success or failure of a task.

Users and customers are encouraged to report defects, or feature requests at Stonebranch Support Desk.

Document References

This document references the following documents:

Document Link	Description
Universal Templates	User documentation for creating, working with, and understanding Universal Templates and Integrations.
Universal Tasks	User documentation for creating Universal Tasks in the Universal Controller user interface.

Changelog

ue-aws-emr-1.0.0 (2024-06-

07

06)

Initial Version

Versions Compared

Old Version 2

New Version 3

Key

Overview

Import Universal Template

Configure Universal Task

Integration Modifications

Document References

Changelog

ue-aws-emr-1.0.0 (2024-06-

06)

Page Comparison

Versions Compared

Old Version 2

New Version 3

Key

Overview

Import Universal Template

Configure Universal Task

Integration Modifications

Document References

Changelog

ue-aws-emr-1.0.0 (2024-06-

06)