Disclaimer

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at  Stonebranch Integration Hub - Terms of Use.

Introduction

This Universal Task allows Stonebranch users to perform end-to-end Orchestration and Automation of Jobs & Clusters in Databricks environment, either in AWS or Azure.

Overview

This integration will use the Databricks URL and the user bearer token to connect with the Databricks environment. Users can perform the following actions with respect to the Databricks jobs.

• Create and list jobs

• Get job details

• Run now jobs

• Run submit jobs

• Cancel run jobs

Also with respect to Databricks clusters, this integration can perform the following operations.

• Create, start, and restart a cluster

• Terminate a cluster

• Get cluster information

• List clusters

With respect to Databricks DBFS, this Universal Task also provides a feature to upload larger files.

Version

Refer to Changelog for version history information.

Software Requirements

This integration requires a Universal Agent and a Python runtime to execute the Universal Task against a Databricks environment. This Universal Task has been tested with the Azure Databricks environment API version 2.0.

Software Requirements for Universal Template and Universal Task

Requires Python 3.7.6 or higher. Tested with the Universal Agent bundled Python distribution. It requires "requests" python module. 

Software Requirements for Universal Agent

• Universal Agent for Windows x64 Version 7.1 and later with Python options installed

• Universal Agent for Linux Version 7.1 and later with Python options installed

Software Requirements for Universal Controller

• Universal Controller Version 7.1.0.0 and later

Technical Considerations

• This task uses Python module "requests" to make REST API calls to the Databricks environment.

• Databricks URL and user bearer token would be required as basic input for this Universal Task.

• Authentication is possible either by generating a personal access token in Databricks Environment or an Azure AD-based authentication.

• Using the Azure AD based Authentication involves some configuration as shown below in Azure (https://portal.azure.com).

  • Login to Azure; Use the App registrations services to create an application registration for Azure Databricks to get the client id / Client secret credentials

  • For the newly created Databricks App registration add the API permission for Azure Databricks as shown below:
    
    
    
    

  • Finally, add the service principal to Azure Databricks

• Please refer to the Databricks API related to jobs and clusters.

Key Features

Import Universal Template

To use this downloadable Universal Template, you first must perform the following steps:

1. This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.
2. To import the Universal Template into your Controller, follow the instructions here.
3. When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list.

Modifications of this integration, applied by users or customers, before or after import, might affect the supportability of this integration. 

Configure Universal Task

For the new Universal Task type, create a new task, and enter the task-specific details that were created in the Universal Template.

Input Fields

Task Examples

List Job

Run now Job

Run Now Job - Script Details

Run Submit Job

List Cluster

Upload Local File to DBFS

Task Output

Exit Codes

The exit codes for Universal Task are described in the following table.

Document References

This document references the following documents: