Disclaimer

Your use of this download is governed by Stonebranch's Terms of Use.

Overview

Prefect is an open-source workflow management system designed to orchestrate data pipelines in Python. This integration provides the capability to trigger from UAC a Prefect Flow Run and monitor its execution providing useful metadata.

Key Features

Feature	Description
Launch Flow Runs	Launch Flow Runs on Prefect Cloud and monitor until the flow gets completed with success or failure. Flow Run metadata can be provided as part of Extension Output.

Version Information

Template Name	Extension Name	Version	Status
Prefect	ue-prefect	1.0.0	Fixes and new Features are introduced.

Refer to Changelog for version history information.

Software Requirements

This integration requires a Universal Agent and a Python runtime to execute the Universal Task.

Area	Details
Python Version	It requires Python 3.11.
Universal Agent Compatibility	Compatible with Universal Agent for Windows x64 and version >= 7.6.0.0. Compatible with Universal Agent for Linux and version >= 7.6.0.0.
Universal Controller Compatibility	Universal Controller Version >= 7.6.0.0.
Network and Connectivity	Network connectivity towards Prefect Cloud is required.
Prefect Cloud and Rest API versions	Compatible with Prefect Cloud version 2.18.1 with REST API version 0.8.4 . It should be compatible on later versions as long as backwards compatibility is guaranteed on the API level.

Supported Actions

Action: Prefect Cloud - Launch Flow Run

This action allows to launch a Flow Run on Prefect Cloud. The task author has to provide the required Credentials, the Workspace ID and the Deployment ID that relates to the Flow and some additional parameters related to the flow execution (more information can be found on the Fields Description section). Task author can decide whether the task will just launch the flow, or in addition wait until a final success or failure state is reached. The last execution Flow Run metadata can be published as part of the Extension Output, while the Flow Run ID, the Flow Run Name and the Flow Run State can be visible directly as output fields during task instance execution.

Configuration examples


Simple Launch Flow Execution using Workspace ID or Name and Deployment ID or Name as dynamic choice fields to retrieve the necessary names from the Prefect Cloud Platform.	Simple Launch Flow Execution using Workspace ID or Name and Deployment ID or Name as IDs that can be copy pasted from Prefect Console. For example purposes, dummy IDs are used.

Launch Flow by providing Flow Parameters and Tags. In this case it is assumed that the Flow expects a Boolean and an integer parameter.	Launch Flow and wait until the Flow ends in Success or in Failure. The latest Flow Run metadata is configured to be included as part of the Extension Output.

Action Output

Output Type

Description

Examples

EXTENSION

The extension output provides the following information:

exit_code, status_description: General info regarding the task execution. For more information users can refer to the exit code table.
invocation.fields: The task configuration used for this task execution.
result.flow_run_info: The latest Flow Run information provided by Prefect during the execution of the task. Populated in case task is configured to publish this information on Extension Output.
result.errors: List of errors that might have occurred during execution.

Successful Scenario with latest Flow Run metadata as part of the Extension Output

{
    "exit_code": 0,
    "status_description": "Task executed successfully.",
    "invocation": {
        "fields": {...}
    },
    "result": {
        "flow_run_info": {
            "id": "c5feefe0-e932-4f1e-9055-03155561938d",
            "created": "2024-04-30T09:41:40.633550+00:00",
            "updated": "2024-04-30T09:41:40.628595+00:00",
            "name": "mydummyname",
            "flow_id": "6342cf22-2baa-4b4f-8c7f-85d731055af8",
            "state_id": "70736b0e-75e2-4156-8bca-8a21f4c538e4",
            "deployment_id": "ef8baf82-a14b-472f-9f8d-5424b66f232d",
            "work_queue_id": null,
            "work_queue_name": null,
            "flow_version": null,
            "deployment_version": "49e24154c00009d471e26e915271e743",
            "parameters": {
                "fail": false,
                "sleeptime": 5
            },
            "idempotency_key": null,
            "context": {
                "context1": "value1",
                "context2": "value2"
            },
            "empirical_policy": {
                "max_retries": 0,
                "retry_delay_seconds": 0,
                "retries": null,
                "retry_delay": null,
                "pause_keys": [],
                "resuming": false
            },
            "tags": [
                "telis 1",
                "telis 2",
                "testing"
            ],
            "parent_task_run_id": null,
            "state_type": "SCHEDULED",
            "state_name": "Scheduled",
            "run_count": 0,
            "expected_start_time": "2024-04-30T09:41:40.633473+00:00",
            "next_scheduled_start_time": "2024-04-30T09:41:40.633473+00:00",
            "start_time": null,
            "end_time": null,
            "total_run_time": 0,
            "estimated_run_time": 0,
            "estimated_start_time_delta": 0.049332,
            "auto_scheduled": false,
            "infrastructure_document_id": null,
            "infrastructure_pid": null,
            "created_by": {
                "id": "34e9ed52-f547-408e-b53d-02968ba97d64",
                "type": "USER",
                "display_value": "theta457-edison"
            },
            "work_pool_id": null,
            "work_pool_name": null,
            "state": {
                "id": "70736b0e-75e2-4156-8bca-8a21f4c538e4",
                "type": "SCHEDULED",
                "name": "Scheduled",
                "timestamp": "2024-04-30T09:41:40.633516+00:00",
                "message": null,
                "data": null,
                "state_details": {
                    "flow_run_id": "c5feefe0-e932-4f1e-9055-03155561938d",
                    "task_run_id": null,
                    "child_flow_run_id": null,
                    "scheduled_time": "2024-04-30T09:41:40.633473+00:00",
                    "cache_key": null,
                    "cache_expiration": null,
                    "untrackable_result": false,
                    "pause_timeout": null,
                    "pause_reschedule": false,
                    "pause_key": null,
                    "run_input_keyset": null,
                    "refresh_cache": null,
                    "retriable": null,
                    "transition_id": null,
                    "task_parameters_id": null
                }
            },
            "job_variables": {
                "var1": "value1",
                "var2": "value2"
            }
        },
        "errors": []
    }
}

Flow Reached in a Failed State