Disclaimer

Your use of this download is governed by Stonebranch’s Terms of Use, which are available at https://www.stonebranch.com/integration-hub/Terms-and-Privacy/Terms-of-Use

Version Information

Overview

Fivetran is a cloud-based data integration platform that helps businesses automate the process of extracting data from various sources, transforming it, and loading it into a data warehouse for analysis. It offers a wide range of pre-built connectors to popular data sources such as databases, marketing platforms, CRMs, and more. Fivetran's connectors are designed to be easy to set up and use, allowing businesses to focus on data analysis rather than data integration. The platform also offers features such as data transformation, scheduling, monitoring, and access control to help organizations manage their data pipelines more efficiently. Fivetran supports a variety of data warehouses, including Amazon Redshift, Google BigQuery, Snowflake, and Microsoft Azure.

This integration allows to create and execute Fivetran Tasks in Universal Controller.

Key Features

Requirements

This integration requires a Universal Agent and a Python runtime to execute the Universal Task.

Supported Actions

Action: “Sync”

Triggers a data sync for an existing connector within your Fivetran account without waiting for the next scheduled sync. If the connector is paused, the data sync will be scheduled to be performed when the connector is re-enabled.

Task definition author has the capability to configure the task to wait for success or failure, through Wait for Sync to Finish field.

Configuration examples

User Scenario: Trigger Connector Sync	User Scenario: Trigger Connector Sync and Wait Until Success or Failure

Action Output

{
    "exit_code": 0,
    "status_description": "Task executed successfully.",
     "invocation": {
	"extension": "ue-fivetran",
	"version": "1.0.0",
        "fields": { ... },
    },
    "result": {
        "latest_connector_details": {
            "destination_schema": {
                "name": "github"
            },
            "id": "paving_diffuser",
            "group_id": "fistula_cheek",
            "connector_type_id": "github",
            "name": "github",
            "connected_by": "balsam_dilapidated",
            "created_at": "2023-05-23T12:51:13.378309Z",
            "succeeded_at": "2023-05-24T15:33:12.153Z",
            "failed_at": "2023-05-23T13:09:32.747Z",
            "paused": false,
            "pause_after_trial": false,
            "version": 0,
            "sync_frequency": 360,
            "schedule_type": "auto",
            "status": {
                "setup_state": "connected",
                "schema_status": "ready",
                "sync_state": "scheduled",
                "update_state": "on_schedule",
                "is_historical_sync": false,
                "tasks": [],
                "warnings": []
            },
            "config": {
                "sync_mode": "SPECIFIC_REPOSITORIES",
                "repositories": [
                    "ue-fivetran/test-repo-ue-fivetran"
                ],
                "auth_mode": "PERSONAL_ACCESS_TOKEN",
                "username": "fivetran.user@dev.com",
                "pat": "******",
                "use_webhooks": true
            }
        }
    }
}

{
    "exit_code": 31,
    "status_description": "Task execution failed.",
    "invocation": {
        "extension": "ue-fivetran",
	"version": "1.0.0",
        "fields": { ... },
    },
    "result": {
        "errors": [
            "Invalid authorization credentials"
        ],
        "execution_info": {
            "url": "https://api.fivetran.com/v1/groups",
            "http_code": 401
        }
    }
}

Αction: “Re-sync”

Triggers a full historical data sync for a specific connector. If the connector is paused, the data sync will be scheduled to be performed when the connector is re-enabled. Task definition author has the capability to configure the task to wait for success or failure, through Wait for Sync to Finish field.

Configuration examples

User Scenario: Trigger Connector Re-sync

Action Output

{
    "exit_code": 0,
    "status_description": "Task executed successfully.",
    "invocation": {
        "extension": "ue-fivetran",
        "version": "1.0.0",
        "fields": { ... },
    },
    "result": {
        "latest_connector_details": {
            "code": "Success",
            "data": {
                "destination_schema": {
                    "name": "github"
                },
                "id": "paving_diffuser",
                "group_id": "fistula_cheek",
                "connector_type_id": "github",
                "name": "github",
                "connected_by": "balsam_dilapidated",
                "created_at": "2023-05-23T12:51:13.378309Z",
                "succeeded_at": "2023-05-25T14:09:37.460Z",
                "failed_at": "2023-05-24T15:34:22.341Z",
                "paused": false,
                "pause_after_trial": false,
                "version": 0,
                "sync_frequency": 360,
                "schedule_type": "auto",
                "status": {
                    "setup_state": "connected",
                    "schema_status": "ready",
                    "sync_state": "scheduled",
                    "update_state": "on_schedule",
                    "is_historical_sync": false,
                    "tasks": [],
                    "warnings": []
                },
                "config": {
                    "sync_mode": "SPECIFIC_REPOSITORIES",
                    "repositories": [
                        "ue-fivetran/test-repo-ue-fivetran"
                    ],
                    "auth_mode": "PERSONAL_ACCESS_TOKEN",
                    "username": "fivetran.user@dev.com",
                    "pat": "******",
                    "use_webhooks": true
                }
            }
        }
    }
}

{
    "exit_code": 1,
    "status_description": "Task execution failed.",
    "invocation": {
        "extension": "ue-fivetran",
        "version": "1.0.0",
        "fields": { ...
        },
    },
    "result": {
        "errors": [
            "Invalid authorization credentials"
        ],
        "execution_info": {
            "url": "https://api.fivetran.com/v1/groups",
            "http_code": 401
        }
    }
}

Input Fields

Output-Only fields

Extension Cancelation and Re-run

Cancellation from UC will just stop the execution of the Task. The Re-run of the task behaves the same way as on the initial run.

• In case Action = “Sync” and in case Force Sync = “True” and the connector is currently syncing, sync is stopped and restarted. In the case where Force Sync = “False”, the connector will sync only if it isn't currently syncing.

• In case Action = “Re-sync (Historical Sync)” and the connector is currently syncing, sync is always stopped and restarted (based on the logic of Fivetran API)

In all cases, if Wait for Sync to Finish = “True” the Task will wait until the Sync Status reaches Success or Failure.

Exit Codes

STDOUT and STDERR

How To

Import Universal Template

To use the Universal Template, you first must perform the following steps.

1. This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.

2. To import the Universal Template into your Controller, follow these instructions.

3. When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list.

Modifications of this integration, applied by users or customers, before or after import, might affect the supportability of this integration. For more information refer to Integration Modifications.

Configure Universal Task

For a new Universal Task, create a new task, and enter the required input fields.

Integration Modifications

Modifications applied by users or customers, before or after import, might affect the supportability of this integration. The following modifications are discouraged from retaining the support level as applied for this integration.

• Python code modifications should not be done.

• Template Modifications

  • General Section

    • "Name", "Extension", "Variable Prefix", and "Icon" should not be changed.

  • Universal Template Details Section

    • "Template Type", "Agent Type", "Send Extension Variables", and "Always Cancel on Force Finish" should not be changed.

  • Result Processing Defaults Section

    • Success and Failure Exit codes should not be changed.

    • Success and Failure Output processing should not be changed.

  • Fields Restriction Section
    The setup of the template does not impose any restrictions, However with respect to the "Exit Code Processing Fields" section.

    1. Success/Failure exit codes need to be respected.

    2. In principle, as STDERR and STDOUT outputs can change in follow-up releases of this integration, they should not be considered as a reliable source for determining the success or failure of a task.

Users and customers are encouraged to report defects, or feature requests at Stonebranch Support Desk.

Document References

This document references the following documents:

Known Issues

Fivetran does not provide specific APIs to retrieve the status of a specific Sync Instance, therefore identification of a Successful or Failed completion, is implemented by means of a “Best Effort” algorithm. This algorithm relies on the fact that immediately after the Sync is triggered successfully, succeeded_at & failed_at timestamps are retrieved and saved. During polling, succeeded_at or failed_at timestamps are getting compared with the saved values. Polling will continue until the succeeded_at or the failed_at are updated which indicates either a successful or a failed sync. This logic is expected to work correctly in the vast majority of cases if not all. However, there could be some corner cases where it might not work as expected.

Changelog

ue-fivetran-1.0.1 (2023-10-18)

Fixes

• Fixed: Execute the integration with a Python installation, other than the Agent-bundled one (#34545).

ue-fivetran-1.0.0 (2023-06-08)

Initial Version