Disclaimer
Your use of this download is governed by Stonebranch’s Terms of Use, which are available at https://www.stonebranch.com/integration-hub/Terms-and-Privacy/Terms-of-Use/
Introduction
This Universal Task allows Stonebranch users to perform end-to-end Orchestration and Automation of Jobs & Clusters in Databricks environment, either in AWS or Azure.
Overview
This task will use the Databricks URL and the user bearer token to connect with the Databricks environment.
Users can perform the following with respect to the Databricks jobs.
Create and list jobs
Get job details
Run now jobs
Run submit jobs
Cancel run jobs
Also with respect to Databricks clusters, this Universal Task can perform the following operations:
Create, start and restart a cluster
Terminate a cluster
Get a cluster info
List clusters
With respect to Databricks DBFS , this Universal Task also provides a feature to upload larger files.
Software Requirements
This integration requires a Universal Agent and a Python runtime to execute the Universal Task against a Databricks environment.
Software Requirements for /wiki/spaces/UC71x/pages/5178050 and /wiki/spaces/UC71x/pages/5180675
Requires Python 3.6 or higher. Tested with the Universal Agent bundled Python distribution.
Python modules required
requests
Software Requirements for Universal Agent
Universal Agent for Windows x64 Version 6.6 and later with Python options installed
Universal Agent for Linux Version 6.6 and later with Python options installed
Software Requirements for Universal Controller
Universal Controller Version 6.6.0.0 and later
Software Requirements for the Application to be Scheduled
This Universal Task has been tested with the Azure Databricks environment -API version 2.0.
Technical Considerations
This task uses Python modules requests to make REST-API calls to the Databricks environment.
Databricks URL and user bearer token would be required as basic input for this Universal Task.
Please refer to the Databricks API related to jobs and clusters in URL: https://docs.Databricks.com/dev-tools/api/latest/
Key Features
Feature | Description |
Create Job | Create a job in a Databricks environment from Universal Controller. Here, a JSON input for job creation in Databricks environment will be used. |
List jobs | List the jobs available within the Databricks environment. |
Get Job details | Provides an existing job definition in Databricks by providing the job ID as input. |
Run now Jobs | This feature helps to run an existing job in Databricks environment using the run time input parameters supplied in JSON from the universal task and the Universal Controller will be monitoring the execution of the job until it gets completed. |
Run Submit jobs | This feature helps to run a job in Databricks environment that can be dynamically defined in JSON as an input parameter in the Universal Task and the Universal Controller will be monitoring the execution of the job until it gets completed. |
Cancel Run job | Cancel a execution of job that is in running state within the Databricks environment. |
Create Cluster | Create a cluster in Databricks environment. Input to be provided in the JSON in a script in this Universal Task. |
List clusters | List the clusters available in the Databricks environment. |
Start cluster | Start a cluster that is in stopped state in Databricks. |
Restart cluster | Restart a cluster in the Databricks environment. |
Terminate cluster | Terminate cluster in Databricks environment by providing cluster ID as input. |
Get a Cluster info | Provides the definition of an existing cluster in Databricks environment in JSON. |
Upload file to DBFS | Upload a file from local server to a Databricks file system DBFS. |
Import Databricks Integration Downloadable Universal Template
To use this downloadable Universal Template, you first must perform the following steps:
This Universal Task requires the /wiki/spaces/UC71x/pages/5178443 feature. Check that the/wiki/spaces/UC71x/pages/5177877 system property has been set to true.
Copy or Transfer the Universal Template file to a directory that can be accessed by the Universal Controller Tomcat user.
In the Universal Controller UI, select Configuration > Universal Templates to display the current list of /wiki/spaces/UC71x/pages/5178054.
Right-click any column header on the list to display an Action menu.
Select Import from the menu, enter the directory containing the Universal Template file(s) that you want to import, and click OK.
When the files have been imported successfully, the Universal Template will appear on the list.
Configure Databricks Integration Universal Task
For the new Universal Task type, create a new task, and enter the task-specific details that were created in the Universal Template.
Field Descriptions for Databricks Universal Task
Field | Description |
Databricks URL | Specify the Databricks URL. |
Bearer Token | Provide the Databricks Personal token or the Azure AD token. |
Databricks Function | Select a Function that would like to perform with Databricks. |
Create Request Script | Feed the script for the new job creation or cluster in Databricks. |
Job ID | Provide the Databricks Job ID. |
Job Run Request | Specify the parameters for Jar or notebook or python or spark-submit or the Job submit run request. |
Run ID | Specify the Databricks Run ID. |
Cluster ID | Provide the cluster ID. |
Local file name | Local file name with path. |
DBFS file name | Provide the Databricks file path and name. |
overwrite | Specify if the uploaded files need to overwritten in DBFS. |
Examples for Databricks Integration Universal Tasks
Run now Job
Run Submit Job
List Cluster
Upload Local File to DBFS
Document References
This document references the following documents:
Name | Location | Description |
---|---|---|
Universal Templates | https://docs.stonebranch.com/confluence/display/UC71x/Universal+Templates | User documentation for creating Universal Templates in the Universal Controller user interface. |
Universal Tasks | https://docs.stonebranch.com/confluence/display/UC71x/Universal+Tasks | User documentation for creating Universal Tasks in the Universal Controller user interface. |