Azure Data Factory: Schedule, Trigger, and Monitor

Owned by Former user (Deleted)
Last updated: Oct 13, 2023 by Stonebranch (Deactivated)
4 min read

Disclaimer

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at https://www.stonebranch.com/integration-hub/Terms-and-Privacy/Terms-of-Use/

Introduction

This Integration allows Stonebranch users to schedule, trigger, and monitor the Azure Data Factory pipeline process directly from Universal Controller.  

Overview

  • This integration uses Python modules azure-mgmt-resource and azure-mgmt-datafactory to make REST-API calls to Azure Data Factory.

  • This integration will use the Azure Tenant id , Subscription id , client id ,client secret, Resource group, and location for authenticating the REST-API calls to Azure Data Factory. 

  • User can perform the following Azure Data Factory operations:

    • Run a Pipeline. 

    • Get information on a Pipeline.

    • List all Pipelines.

    • Cancel Pipeline run.

    • List factory by resource group.

  • Also, with respect to Azure Data Factory triggers, users can perform the following operations from UAC:

    • Start Trigger.

    • Stop Trigger.

    • List Trigger by Factory.

  • UAC also can restart a failed pipeline either from the failed step or from any activity name in the failed pipeline.

Software Requirements

This integration requires a Universal Agent and a Python runtime to execute the Universal Task against an Azure Data Factory.

Software Requirements for Universal Template and Universal Task

  • Requires Python 3.4 or higher. Tested with the Universal Agent bundled Python distribution.

  • Python modules required:

    • azure-mgmt-resource

    • azure-mgmt-datafactory

    • azure.identity

Software Requirements for Universal Agent

Either:

  • Universal Agent for Windows x64 Version 6.9 and later with Python options installed.

  • Universal Agent for Linux Version 6.9 and later with Python options installed.

Software Requirements for Universal Controller

  • Universal Controller Version 6.9.0.0 and later.

Software Requirements for the Application to be Scheduled

This Universal Task has been tested with the Azure Data Factory Version 2.

Technical Considerations

  • This integration uses the Python modules Azure resource Management Module and Azure Data Factory management to make REST-API calls with Azure Data Factory.

  • Use Azure App Registration services to create an app, client ID, and client and associate it with Data Factory to provide appropriate roles (for example, Contributor).

Key Features

Feature

Description

Run a Pipeline

Execute a pipeline defined in Azure Data Factory. Schedule, trigger, and monitor the execution of a pipeline.

Get Information on a Pipeline

Get a information about the pipeline name .

List all Pipelines

List all pipelines that belongs to a Data Factory.

Cancel a Pipeline Run

Abort a Pipeline execution from UAC by providing the run ID for the Pipeline execution.

List Factory by resource group

List the available factory name that belongs to a resource group.

Start a Trigger

Start a trigger that will manage pipeline execution schedule in the Azure Data Factory.

Stop a Trigger

Stop a trigger that is in the Azure Data Factory.

List Trigger by Factory

List all the triggers that belongs to the Data Factory.

Import Azure Data Factory Integration Downloadable Universal Template

To use this downloadable Universal Template, you first must perform the following steps:

  1. This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.
  2. To import the Universal Template into your Controller, follow the instructions here.
  3. When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list.

Configure Azure Data Factory Integration Universal Task

For the new Universal Task type, create a new task, and enter the task-specific details that were created in the Universal Template.

Field Descriptions for Azure Data Factory Integration Universal Task

Field

Description

Azure Location

Provide the Azure location.

Tenant ID

Provide the Azure Tenant ID or Directory ID.

Azure Client ID and Client Secret

Provide the Azure client ID and client secret (Client ID as user name and client secret as the password) - This comes from your Azure App registration services.

Subscription

Azure Subscription ID.

Azure Resource Group

Azure Resource group responsible for the Data Factory.

Data factory Operation

Select the required Azure Data Factory function for the task.

Data Factory Name

Provide the Azure Data Factory Name.

Pipeline Name

Pipeline name that needs to be executed in Azure Data factory.

Pipeline Parameters

Provide the pipeline run time parameters that needs to be passed for the pipeline execution.

Run ID

Run id for the Azure Data Factory Pipeline execution - Required in case of a restart.

Restart Pipeline

Check if the pipeline execution needs to be restarted for a failure.

Start From Failure

Pipeline needs to be restarted from a Failure step.

Start Activity Name

If Start from Failure is not checked, specify if a failed pipeline needs to be restarted from a specific activity.

Run ID for Restart

Provide the Pipeline run ID for the restart.

Trigger Name

Trigger Name in the Azure Data Factory.

Polling

True for ARMPolling, False for no polling, or a polling object for personal polling strategy.

Examples for Azure Data Factory Integration Universal Tasks

Run a Pipeline

Pipeline Execution Logs

Restart a Failed Pipeline Execution

Cancel a Pipeline Run

Other Azure Data Factory Operation from UAC


Document References

This document references the following documents:

Name

Description

Universal Templates

User documentation for creating Universal Templates in the Universal Controller user interface.

Universal Tasks

User documentation for creating Universal Tasks in the Universal Controller user interface.