Your use of this download is governed by Stonebranch’s Terms of Use, which are available at Stonebranch Integration Hub - Terms of Use.

Version Information

Template Name	Extension Name	Extension Version
DBT Core CLI	ue-dbt-core-cli	1.0.

0

1

Refer to Changelog for version history information.

...

This integration provides an interface to call the DBT Core CLI and enables users to execute DBT Core CLI tasks in Universal Controler. This is particularly useful when UC acts as a data pipeline orchestrator and DBT is a step within a data pipeline Universal Controller workflow.

Key Features

Feature	Description
Show Environment Information	Show DBT version and DBT environment variables information as Task Instance STDOUT. Enables users to retrieve Information on the dbt core installation.
Execute Command	Execute a given DBT command. DBT output is displayed as task instance STDOUT.
Credentials stored encrypted and passed as environment variables	Ability to safely store and pass UC Credentials as DBT environment variables.
Display dbt artifacts in Extension Output	Artifacts can be collected, and related information can be published within the Universal Controller for subsequent workflow tasks to use.

Requirements

This integration requires a Universal Agent and a Python runtime to execute the Universal Task.

Area	Details
Python Version	Requires Python of version 3.7. Tested with the Universal Agent bundled Python distribution (python version 3.7.

6

16)
Universal Agent	Both Windows and Linux agents are supported: Universal Agent for Windows x64 Version >= 7.4.0.0 Universal Agent for Linux Version >= 7.4.0.0
Universal Controller	Universal Controller Version >= 7.4.0.0
Network and Connectivity	Connectivity towards Warehouses that DBT uses should be established.
DBT Module	This integration is tested against DBT core module (dbt-core) 1.5.1. It is required to be installed manually on the Agent. For more information refer to dbt-core official installation instructions.

Supported Actions

Action: Get Environment Information

...

No DBT Input Parameters are required to be passed apart from the Action itself. Common input parameters such as Runtime Directory or Environment Variables might come in handy if need to be configured on the task definition.

Configuration example

...

Image Added
User Scenario: Show dbt environment information.

Action Output

Output Type

Description (Successful Execution)

Example (Successful Execution)

EXTENSION

The extension output follows the standard Extension Output format, providing:

“exit_code“, “status“, “status_description“: General info regarding the task execution.
“invocation”
- “fields”: The task configuration used for this task execution.
“result“ fields
- “execution_info”:
  - “command_line”: The dbt command line that is executed
  - “command_exit_code”: Is the dbt command line exit code

Code Block

{
    "exit_code": 0,
    "status": "SUCCESS",
    "status_description": "Task executed successfully.",
    "invocation": {
        "extension": "ue-dbt-core-cli",
        "version": "1.0.0",
        "fields": {
            "action": "Show Environment Information",
            "credentials": null,
            "command_line": null,
            "artifact_content_visibility": "--None--"
        }
    },
    "result": {
        "execution_info": {
            "command_line": "dbt --version",
            "command_exit_code": 0
        }
    }
}

Description (Failed Execution)

Example (Failed Execution)

The extension output follows the standard Extension Output format, providing:

“exit_code“, “status“, “status_description“: Those attributes are set accordingly when an error happens. A list of possible values is provided in the extension exit codes table in this document.
“invocation”
- “fields”: The task configuration used for this task execution.
“result“ fields
- “execution_info”:
  - “command_line”: The dbt command line that is executed and resulted in erroneous execution
  - “command_exit_code”: is the dbt command line exit code if a dbt command is being executed.

Code Block

{
    "exit_code": 1,
    "status": "FAIL",
    "status_description": "Execution Failed: Refer to STDERR or Extension Output for more details.",
    "invocation": {
        "extension": "ue-dbt-core-cli",
        "version": "1.0.0",
        "fields": {
            "action": "Show Environment Information",
            "credentials": null,
            "command_line": null,
            "artifact_content_visibility": "--None--"
        }
    },
    "result": {
        "execution_info": {
            "command_line": "dbt --version",
            "command_exit_code": 1
        }
    }
}

STDOUT

On Extension STDOUT the following information is displayed:

The version of DBT
The DBT environment variables present at the execution time
The runtime directory

Code Block

DBT Core and Plugins Version Information
===========================================

Core:
  - installed: 1.5.1

Plugins:
  - postgres: 1.5.1 - Update available!

  At least one plugin is out of date or incompatible with dbt-core.
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation


DBT Environment Variables
===========================================

+--------------------+---------+
| DBT ENV VARIABLE   | Value   |
+====================+=========+
| DBT_STB_VAR        | stb     |
+--------------------+---------+

Runtime Directory
===========================================
C:\Program Files\Universal\UAGSrv

Action: “Execute Command”

When Action = “Execute Command” then the command provided as Command Line is executed. Environment variables are set for credentials purposes through “Credentials as Environment Variables” before Command Line is executed.

For DBT commands that “target path” is required, it should be set either as an environment variable (DBT_TARGET_PATH) or provided on the Command Line.

Configuration examples

Image Removed

Image Added

Image Added
User Scenario: Execute dbt command using Credentials as DBT Environment Variables. User Scenario: Execute dbt command using Credentials as DBT Environment Variables. The above script shows how to set DBT environment variables using UAC Functions on Resolvable Credentials.

Action Output

Output Type

Description (Successful Execution)

Example (Successful Execution)

EXTENSION

The extension output follows the standard Extension Output format, providing:

“exit_code“, “status“, “status_description“: General info regarding the task execution.
“invocation”
- “fields”: The task configuration used for this task execution.
“result“ fields:
- “execution_info”:
  - “command_line”: The dbt command line that is executed
  - “command_exit_code”: Is the dbt command line exit code
- “artifacts” field: a list of artifact information produced by dbt command. for each element of the list the following attributes are available
  - “type”: the type of the artifact. Manifest and Run Results artifacts are supported.
  - “path”: The path where the artifact file is located.
  - “content”: The content of the artifact file if configured to be printed on Extension Output, else null.

Info

Note on artifacts: The artifacts that are supported are the “Manifest” and the “Run Results”. Both of them are located in the “target” directory. The target directory will be identified automatically by this integration taking into account --target-path CLI flag and DBT_TARGET_PATH environment variable, but not take into account any configuration on dbt_project.yml file, since the configuration of “target” directory path in dbt_project.yml is considered deprecated and expected not to work in dbt core version 1.6.

The precedence order for detection of the target directory is: CLI flag, Environment variable.

Code Block

{
    "exit_code": 0,
    "status": "SUCCESS",
    "status_description": "Task executed successfully.",
    "invocation": {
        "extension": "ue-dbt-core-cli",
        "version": "1.0.0",
        "fields": {
            "action": "Execute Command",
            "credentials": "/home/agent/tmp/e8917d08-0341-4239-a486-f53570237817",
            "command_line": "dbt run
--profiles-dir=/home/

.../

dbt_workspace/.dbt
--project-dir=/home/

.../dbt

dbt_workspace/my_stb_dbt_project 
--target-path=/home

/...

/dbt_workspace/my_stb_dbt_project/ue_execution",
            "artifact_content_visibility": "--None--"
        }
    },
    "result": {
        "execution_info": {
            "command_line": "dbt run --profiles-dir=/home

/...

/dbt_workspace/.dbt 
--project-dir=/home

/...

/dbt_workspace/my_stb_dbt_project 
--target-path=/home

/ue-dev/USERS/ioanna

/dbt_workspace/my_stb_dbt_project/ue_execution",
            "command_exit_code": 0,
            "artifacts": [
                {
                    "type": "manifest",
                    "path": "/home

/...

/dbt_workspace/my_stb_dbt_project/ue_execution/manifest.json",
                    "content": null
                },
                {
                    "type": "run_results",
                    "path": "/home/.../dbt_workspace/my_stb_dbt_project/ue_execution/run_results.json",
                    "content": null
                }
            ]
        }
    }
}

Description (Failed Execution)

Example (Failed Execution)

The extension output follows the standard Extension Output format, providing:

“exit_code“, “status“, “status_description“: Will be set according to the specific error that happened during the execution. A list of possible values is provided in the extension exit codes table in this document.
“invocation”
- “fields”: Same as in successful execution
“result“ fields:
- “execution_info”:
  - “command_line”: The dbt command line that is executed
  - “command_exit_code”: Is the dbt command line exit code
- “artifacts” field: a list of artifact information produced by dbt command. for each element of the list, the following attributes are available
  - “type”: the type of the artifact. Manifest and Run Results artifacts are supported.
  - “path”: The path where the artifact file is located.
  - “content”: The content of the artifact file if it is configured to be printed on Extension Output, else null.

Code Block

{
    "exit_code": 1,
    "status": "FAIL",
    "status_description": "Execution Failed: Refer to STDERR or Extension Output for more details.",
    "invocation": {
        "extension": "ue-dbt-core-cli",
        "version": "1.0.0",
        "fields": {
            "action": "Execute Command",
            "credentials": "/home/agent/tmp/587a68dd-3220-41e8-94b2-a2d0caf69445",
            "command_line": "dbt build --profiles-dir=/home/

.../

dbt_workspace/.dbt 
--project-dir=/home

/...

/dbt_workspace/my_stb_dbt_project 
--target-path=/home

/...

/dbt_workspace/my_stb_dbt_project/ue_execution",
            "artifact_content_visibility": "Run Results on Extension Output",
        }
    },
    "result": {
        "execution_info": {
            "command_line": "dbt build --profiles-dir=/home

/...

/dbt_workspace/.dbt 
--project-dir=/home/

.../

dbt_workspace/my_stb_dbt_project 
--target-path=/home/

.../

dbt_workspace/my_stb_dbt_project/ue_execution",
            "command_exit_code": 2,
            "artifacts": []
        }
    }
}

STDOUT

The standard dbt command stdout appears also on the STDOUT of the Universal Task Instance. The level of information depends on the Log Level selected at Univesal Task Definition.

Code Block

12:14:02  Running with dbt=1.5.0
12:14:03  Found 2 models, 4 tests, 0 snapshots, 0 analyses, 307 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics, 0 groups
12:14:03
12:14:04  Concurrency: 1 threads (target='dev')
12:14:04
12:14:04  1 of 2 START sql table model dev_schema.my_first_dbt_model ..................... [RUN]
12:14:05  1 of 2 OK created sql table model dev_schema.my_first_dbt_model ................ [SELECT 2 in 0.81s]
12:14:05  2 of 2 START sql view model dev_schema.my_second_dbt_model ..................... [RUN]
12:14:06  2 of 2 OK created sql view model dev_schema.my_second_dbt_model ................ [CREATE VIEW in 0.81s]
12:14:06
12:14:06  Finished running 1 table model, 1 view model in 0 hours 0 minutes and 3.97 seconds (3.97s).
12:14:06
12:14:06  Completed successfully
12:14:06
12:14:07  Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2

...

Input Fields

Field	Type	Default Value	Mandatory	Description	Introduced in Version
Action	Choice	Show Environment Information	Yes	The action performed upon the task execution. Available options: Show Environment Information Execute Command	1.0.0
Credentials as Environment Variables	Credentials		Optional	A script that allows to set Environment Variables that are used as credentials inside DBT The format of this script should be `MY_CREDENTIAL_ENV_VARIABLE_A=<UC credential function call A> MY_CREDENTIAL_ENV_VARIABLE_B=<UC credential function call B> ...` Available when Action = “Execute Command”	1.0.0
Artifact Content Visibility	Choice	--None--	Optional	--None-- Run Results on Extension Output Available when Action = “Execute Command”	1.0.0
Command Line	Large Text		Optional	The full command line. Available when Action = “Execute Command”	1.0.0

Output-Only fields

No output only fields

...

Cancellation from UC will just stop the execution of the Task. The Re-run of the task behaves the same way as on the initial run.

...

Exit Codes

Extension Exit Code	Status	Status Description	Meaning
0	SUCCESS	“Task executed successfully“	Successful Execution, in case DBT exit code is 0.
1	FAIL	“Execution Failed: <error details>”	Failed Execution, in case DBT exit code is other than 0.
20	DATA_VALIDATION_ERROR	“Data Validation Error: <error details>.“	Input data validation error.

STDOUT and STDERR

STDOUT and STDERR provide additional information to User. The populated content can be changed in future versions of this extension without notice. Backward compatibility is not guaranteed.

Depending on the log level the Universal Task is started by the controller, the DBT_LOG_LEVEL environment variable is set prior to the execution of the DBT command based on the mapping below.

Universal Task Log Level	DBT Log Level
Trace & Debug	Debug
Info	info
Warn	warn
Error & Severe	error

The following applies:

IF DBT_LOG_LEVEL is already set then it will not be overwritten.
IF --log-level command line option is used, by default dbt uses that value from the command line

...

Modifications of this integration, applied by users or customers, before or after import, might affect the supportability of this integration. For more information refer to Integration Modifications.

Configure Universal Task

For a new Universal Task, create a new task, and enter the required input fields.

Anchor

	IntegrationModifications
	IntegrationModifications

Integration Modifications

...

This document references the following documents:

Document Link	Description
Universal Templates	User documentation for creating, working with and understanding Universal Templates and Integrations.
Universal Tasks	User documentation for creating Universal Tasks in the Universal Controller user interface.

Anchor

	changelog
	changelog

Changelog

ue-dbt-core-cli-1.0.1 (2023-10-18)

Fixes

Fixed: Include integration icon on Universal Template (#34228).

ue-dbt-core-cli-1.0.0(2023-08-16)

Initial Version

Versions Compared

Old Version 2

New Version Current

Key

Version Information

Key Features

Requirements

Supported Actions

Action: Get Environment Information

Configuration example

Image Added
User Scenario: Show dbt environment information.

Action Output

Action: “Execute Command”

Configuration examples

Image Added

Image Added
User Scenario: Execute dbt command using Credentials as DBT Environment Variables. User Scenario: Execute dbt command using Credentials as DBT Environment Variables. The above script shows how to set DBT environment variables using UAC Functions on Resolvable Credentials.

Action Output

Input Fields

Output-Only fields

Exit Codes

STDOUT and STDERR

Configure Universal Task

Integration Modifications

Changelog

ue-dbt-core-cli-1.0.1 (2023-10-18)

Fixes

ue-dbt-core-cli-1.0.0(2023-08-16)

Page Comparison

Versions Compared

Old Version 2

New Version Current

Key

Version Information

Key Features

Requirements

Supported Actions

Action: Get Environment Information

Configuration example

Image AddedUser Scenario: Show dbt environment information.

Action Output

Action: “Execute Command”

Configuration examples

Image AddedImage AddedUser Scenario: Execute dbt command using Credentials as DBT Environment Variables.User Scenario: Execute dbt command using Credentials as DBT Environment Variables. The above script shows how to set DBT environment variables using UAC Functions on Resolvable Credentials.

Action Output

Input Fields

Output-Only fields

Exit Codes

STDOUT and STDERR

Configure Universal Task

Integration Modifications

Changelog

ue-dbt-core-cli-1.0.1 (2023-10-18)

Fixes

ue-dbt-core-cli-1.0.0(2023-08-16)

Image Added
User Scenario: Show dbt environment information.

Image Added

Image Added
User Scenario: Execute dbt command using Credentials as DBT Environment Variables. User Scenario: Execute dbt command using Credentials as DBT Environment Variables. The above script shows how to set DBT environment variables using UAC Functions on Resolvable Credentials.