Disclaimer

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at https://www.stonebranch.com/integration-hub/Terms-and-Privacy/Terms-of-Use/

Overview

This Universal Task provides the capability to run, monitor, and re-start Azure Synapse Pipelines from the Universal Automation Center.

To run a Notebook, the Universal Task can trigger an Azure Synapse pipeline with notebook activity.   

The following main functionalities are supported:

• Run a Pipeline.
• Run a Pipeline with parameters.
• List all Pipelines in a Workspace.
• Monitor the started Synapse Pipeline.
• Cancel a Pipeline Run.
• Cancel a Pipeline Run Recursive.
• Rerun a Pipeline from a specified activity or the beginning.
• Service Principal-based authentication to Azure Synapse.
• Certificate-based TLS connection 

Version Information

Refer to Changelog for version history information.

Software Requirements

This integration requires a Universal Agent and a Python runtime to execute the Universal Task.

Software Requirements for Universal Template and Universal Task

Requires Python 3.7. Tested with the Universal Agent bundled Python distribution.

Software Requirements Universal Agent

Both Windows and Linux agents are supported:

• Universal Agent for Windows x64 Version 7.2.0.0 and later with python options installed.

• Universal Agent for Linux Version 7.2.0.0 and later with python options installed.

Software Requirements Universal Controller

Universal Controller Version 7.2.0.0 and later.

Network and Connectivity Requirements

• Universal Agent host should be able to establish a connection with the Azure Synapse API EndPoint: https://<workspacename>.dev.azuresynapse.net
• Universal Agent host should be able to establish a connection with the Azure Oauth2 API EndPoint: https://login.microsoftonline.com/<tenant_id>/oauth2/token

Key Features

This Universal Task provides the following main features:

Actions

• Creates a run of a pipeline. This action is also used to re-run a pipeline.

• Get Pipelines by Workspace. This action lists all Pipelines in a Workspace.

Instance Commands

Commands can be executed during run-time of the Azure Pipeline

• Cancel Pipeline Run.
• Cancel Pipeline Run Recursive.

Authentication

• Service Principal based authentication is used to connect to the Azure Synapse REST APIs (Application Programming Interface).

• Service Principal based Authentication uses the following information to acquire a Rest API Access token:

  • The Azure Active Directory (tenant) ID where the application is registered.
  • The Application (client) ID for the application registered in Azure AD.
  • The Value of the client secret for the application registered in Azure AD.

  This information needs to be provided as prerequisites to configure a new Azure Synapse Universal Task.

Logging

• The current pipeline status is monitored in a configurable poll interval; e.g., every 60s

Import Universal Template

To use the Universal Template, you first must perform the following steps.

1. This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.

2. To import the Universal Template into your Controller, follow the instructions here.

3. When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list.

Modifications of this integration, applied by users or customers, before or after import, might affect the support of this integration. For more information, refer to Integration Modifications.

Configure Universal Task

For a new Universal Task, create a new task, and enter the required input fields.

Input Fields

The input fields for this Universal Extension are described in the following table.

{
    "PARM1": "hello",
    "PARM2": "world"
}

Output-Only fields

Dynamic Commands

Dynamic commands enable users to interact with a running task instance. The following dynamic commands are supported

Task Example

Create Pipeline Run with Parameters

The following Task starts an Azure Synapse Pipeline, with Basic Pipeline Parameters. 

Azure Synapse Credential Details:

Azure Synapse Basic Pipeline Parameters:

Azure Synapse Basic Pipeline Parameters in Azure Synapse Studio:

When using Pipeline Parameters in the Universal Task, those Parameters need to be set as dynamic Parameters in Azure Synapse Studio.

Example: @pipeline().parameters.note

Finished Task

Once the execution of the Pipeline in Azure is finished, the Task goes to "Success" and the Pipeline Status field shows "Succeeded".

Re-start a Pipeline Run

The following example shows a re-start of a pipeline from the activity PipelineSBDemo3-2.

The re-start is performed in recovery mode, which means that the referenced pipeline run, and the new run are grouped under the same groupId.

Task Output

The Task output provides detailed Information on the processing of the Pipeline.

Exit Codes

The exit codes and related Task Output for this Universal Task are described in the following table.

STDOUT and STDERR provide additional information to User. The populated content can be changed in future versions of this Universal Task without notice. Backward compatibility is not guaranteed.

Integration Modifications

Modifications applied by users or customers, before or after import, might affect the supportability of this integration. The following modifications are discouraged to retain the support level as applied for this integration.

• Python code modifications should not be done.
• Template Modifications
  • General Section
    • "Name", "Extension", "Variable Prefix", "Icon" should not be changed.
  • Universal Template Details Section
    • "Template Type", "Agent Type", "Send Extension Variables", "Always Cancel on Force Finish" should not be changed.
  • Result Processing Defaults Section
    • Success and Failure Exit codes should not be changed.
    • Success and Failure Output processing should not be changed.
  • Fields Restriction Section
    The setup of the template does not impose any restrictions, however with respect to "Exit Code Processing Fields" section.
    1. Success/Failure exit codes need to be respected.
    2. In principle, as STDERR and STDOUT outputs can change in follow-up releases of this integration, they should not be considered as a reliable source for determining success or failure of a task.

Users and customers are encouraged to report defects, or feature requests at Stonebranch Support Desk.

Document References

This document references the following documents:

Changelog

ue-cs-azure-synapse-1.2.0 (2024-05-29)

Enhancements

• Json Output is provided for further processing of the output in, e.g. Workflows.

ue-cs-azure-synapse-1.1.2 (2024-04-25)

Enhancements

• Suport for dynamic Bearer Token

ue-cs-azure-synapse-1.1.0

Enhancements

• Suport for TLS Certificate based connections using either a local certificate file or the default Agent Python certify module certificate file

ue-cs-azure-synapse-1.0.1 (2023-08-24)

Enhancements

• Added: Parameter Support

ue-cs-azure-synapse-1.0.0

Initial Version