Versions Compared

Old Version 3

changes.mady.by.user Stonebranch

Saved on

compared with

New Version Current

changes.mady.by.user Stonebranch

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


Table of Contents
indent5

...

Refer to Changelog for version history information.

Overview

Amazon EMR (Elastic MapReduce) is a cloud-based service offered by Amazon Web Services (AWS) designed to process and analyze large data sets quickly and cost-effectively. Amazon EMR notebooks. AWS EMR Notebooks are Jupyter-based notebooks that are integrated with Amazon EMR, providing execution capabilities and in addition an interactive environment for data exploration and analysis. 

This integration provides the ability the start an AWS EMR notebook execution from UAC and optionally monitor it's execution. It also allows Task authors to specify a large variety of notebook execution configuration options.

...

Feature

Description

Start Notebook Execution

Start an AWS EMR Notebook execution with related parameters


Software Requirements

This integration requires a Universal Agent and a Python runtime to execute the Universal Task.

...

Name

Type

Description

Version Information

Action

Choice

The action performed upon the task execution. Available options:

  • Start Notebook Execution (default)

Default value is “Start Notebook Execution”. Note that “Notebook” is an entity on AWS that later was renamed to “Workspace”. However in AWS documentation both terms are used.

1.0.0

Region

Text

Region for the Amazon Web Service i.e. "us-east-1"

When AWS Region is not populated as part of the task definition, during task execution the integration will look for AWS Region on the task execution environment (AWS configuration file or through AWS environment variables).

1.0.0

Credentials

Credentials

The Credentials definition should be as follows.

  • AWS Access Key ID as "Runtime User".

  • AWS Secret Access Key as "Runtime Password".

When AWS Credentials is not populated as part of the task definition, during task execution the integration will look for AWS Credentials on the task execution environment. Refer to AWS Credential Configuration Options for more information

1.0.0

Assume Role ARN

Large Text

Role ARN of the AWS Assume Role functionality. The Assume Role functionality in AWS (Amazon Web Services) allows a user or service to take on the permissions of another IAM (Identity and Access Management) role temporarily.

If field is left empty Role Assumption is notperformed.

1.0.0

Service Role

Text

The name or ARN of the IAM role that is used as the service role for Amazon EMR (the Amazon EMR role) for the notebook execution.

1.0.0

Execution Engine ID

Text

The unique identifier of the execution engine. For an EMR cluster, this is the cluster ID.

1.0.0

Editor ID

Text

The unique identifier of the EMR Notebook to use for notebook execution.

1.0.0

Relative Path

Text

The path and file name of the notebook file for this execution, relative to the path specified for the EMR Notebook.

For example, if you specify a path of "s3://MyBucket/MyNotebooks" when you create an EMR Notebook for a Notebook with an ID ofe-ABCDEFGHIJK1234567890ABCD (the Editor ID of mentioned above), and you specify a Relative Path of "my_notebook_executions/notebook_execution.ipynb" , the location of the file for the notebook execution is "s3://MyBucket/MyNotebooks/e-ABCDEFGHIJK1234567890ABCD/my_notebook_executions/notebook_execution.ipynb"

1.0.0

Notebook Parameters (JSON)

Large Text

Input parameters in JSON format passed to the Amazon EMR Notebook at runtime for execution.

Code Block
titleExample Input
linenumberstrue
collapsetrue
{
"number": 30,
"word": "random text",
"random_dict": {"random_key": "text"}
}


1.0.0

Wait for Completion

Checkbox

A switch that controls whether the UAC task will wait until the Notebook Execution is completed.

Default value is unchecked.

1.0.0

Polling Interval

Integer

The time (in seconds) between retries for getting the status of Notebook Execution.

Mandatory if Wait for Success or Failure is “true”

As a best practice, if the Notebook Execution expected completion duration is long, set the polling Interval to a larger value. A short value will trigger frequent checks towards AWS which in the case of long-duration jobs is inefficient in terms of resources.

Default value is 15.

1.0.0

Max Number of Polls

Integer

Maximum number of polls. Can be used to control the approximate expected duration of the Notebook Execution (in relation also to Polling Interval (sec))

If left empty the UAC Task Instance will poll indefinitely checking whether the Notebook Execution is completed or resulted in failure.

If the Maximum Number of Polls is reached the exit code of the Universal Task Instance is 40.

Available if Wait for Success or Failure is “true”.

1.0.0

Additional Optional Parameters

Checkbox

A switch that controls if any additional optional parameters and configurations should be passed to the notebook execution.

Default value is unchecked.

1.0.0

Notebook Execution Name

Text

An optional name for the notebook execution.

Available if Additional Optional Parameters is “true”.

1.0.0

Provide Additional Configuration As

Choice

Specifies how additional configuration options for EMR Notebook execution can be defined on the UAC Task.

Available options are:

  • – None --

  • As JSON Text (default)

  • As JSON UAC Script

Available if Additional Optional Parameters is “true”.

1.0.0

Additional Configuration (Text)

Large Text

Additional configuration options for the notebook execution.

Code Block
titleDefault Value
linenumberstrue
collapsetrue
{
    "NotebookInstanceSecurityGroupId":"string",
    "ExecutionEngine": {
        "MasterInstanceSecurityGroupId": "string",
        "ExecutionRoleArn": "string"
    },
    "Tags":[
        {
            "Key": "string",
            "Value": "string"
        }
    ],
    "NotebookS3Location":{
        "Bucket": "string",
        "Key": "string"
    },
    "OutputNotebookS3Location":{
        "Bucket": "string",
        "Key": "string"
    },
    "EnvironmentVariables":{
        "string": "string"
    }
}

The default value is a placeholder value meant to serve as a way to inform the user on what the available fields are. It is not meant to be used directly as a default value for an execution of the integration.

Task authors can use the default value provided, as a template.

  • Root JSON elements (for example NotebookInstanceSecurityGroupId) are optional. Consequently if a JSON element is not required, it should be removed.

  • "string" is a placeholder and needs to be updated with the real value. The value needs to be a JSON string format, which means that depending on the value some characters should be escaped accordingly to JSON grammar.

  • Information about the functionality of each field can be found on the Official AWS Guide.

Field is available if Provide Additional Configuration As is “As JSON Text”.

1.0.0

Additional Configuration (Script)

Script

Provides the same functionality as Additional Configuration (Text) option with the added benefit of being reusable by saving the configuration as an UAC script.

Available if Provide Additional Configuration As is “As JSON UAC Script”.

1.0.0

...