Versions Compared

Old Version 2

changes.mady.by.user Stonebranch

Saved on

compared with

New Version Current

changes.mady.by.user Stonebranch

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Template Name

tension Name

Extension Version

Fivetran

ue-fivetran

1.0.01

Overview

Fivetran is a cloud-based data integration platform that helps businesses automate the process of extracting data from various sources, transforming it, and loading it into a data warehouse for analysis. It offers a wide range of pre-built connectors to popular data sources such as databases, marketing platforms, CRMs, and more. Fivetran's connectors are designed to be easy to set up and use, allowing businesses to focus on data analysis rather than data integration. The platform also offers features such as data transformation, scheduling, monitoring, and access control to help organizations manage their data pipelines more efficiently. Fivetran supports a variety of data warehouses, including Amazon Redshift, Google BigQuery, Snowflake, and Microsoft Azure.

...

Area

Details

Python Version

Requires Python of version 3.7.  Tested with the Universal Agent bundled Python distribution (python version 3.7.616)

Universal Agent

Both Windows and Linux agents are supported:

  • Universal Agent for Windows x64 Version 7.1.0.0 and later.

  • Universal Agent for Linux Version 7.1.0.0 and later.

Universal Controller

Universal Controller Version 7.1.0.0 and later.

Network and Connectivity

Outbound internet connectivity towards Fivetran endpoints.

Fivetran REST API Version Compatibility

This Integration is using uses the following APIs and it is compatible with Fivetran as long as Fivetran retains backward compatibility on the REST API level with respect to the following endpoints and versions.

...

Task definition author has the capability to configure the task to wait for success or failure, through Wait for Sync to Finish field.

Configuration examples


Image AddedImage Added
User Scenario: Trigger Connector Sync
Image Removed
User Scenario: Trigger Connector Sync and Wait Until Success or Failure
Image Removed


Action Output


Output Type

Successful Execution

Example

EXTENSION

The extension output follows the standard Extension Output format, providing:

  • “exit_code“, “status“, “status_description“: General info regarding the task execution.

    • “invocation”

      • “fields”: The task configuration used for this task execution.

    • result fields

      • “latest_connector_details": Contains the latest received Connector Details. Information depends on the result set provided by Fivetran API. Consequently, the data contained in "latest_connector_details" can be backward compatible if Fivetran API retains backward compatibility.


Code Block
{
    "exit_code": 0,
    "status_description": "Task executed successfully.",
     "invocation": {
	"extension": "ue-fivetran",
	"version": "1.0.0",
        "fields": { ... },
    },
    "result": {
        "latest_connector_details": {
            "destination_schema": {
                "name": "github"
            },
            "id": "paving_diffuser",
            "group_id": "fistula_cheek",
            "connector_type_id": "github",
            "name": "github",
            "connected_by": "balsam_dilapidated",
            "created_at": "2023-05-23T12:51:13.378309Z",
            "succeeded_at": "2023-05-24T15:33:12.153Z",
            "failed_at": "2023-05-23T13:09:32.747Z",
            "paused": false,
            "pause_after_trial": false,
            "version": 0,
            "sync_frequency": 360,
            "schedule_type": "auto",
            "status": {
                "setup_state": "connected",
                "schema_status": "ready",
                "sync_state": "scheduled",
                "update_state": "on_schedule",
                "is_historical_sync": false,
                "tasks": [],
                "warnings": []
            },
            "config": {
                "sync_mode": "SPECIFIC_REPOSITORIES",
                "repositories": [
                    "ue-fivetran/test-repo-ue-fivetran"
                ],
                "auth_mode": "PERSONAL_ACCESS_TOKEN",
                "username": "fivetran.user@dev.com",
                "pat": "******",
                "use_webhooks": true
            }
        }
    }
}



Failed Execution

Example


  • “exit_code“, “status“, “status_description“: General info regarding the task execution.

  • “invocation”

    • “fields”: The task configuration used for this task execution.

  • result fields

    • “errors“: The errors reported by the extension task during the execution.

    • “execution_info”:

      • “url”: The endpoint that resulted in erroneous execution

      • “http_code”: The http error from the error response if available


Code Block
{


    "exit_code": 31,
    "status_description": "Task execution failed.",
    "invocation": {
        "extension": "ue-fivetran",
	"version": "1.0.0",
        "fields": { ... },
    },
    "result": {
        "errors": [
            "Invalid authorization credentials"
        ],
        "execution_info": {
            "url": "https://api.fivetran.com/v1/groups",
            "http_code": 401
        }
    }
}



Αction: “Re-sync”

Triggers a full historical data sync for a specific connector. If the connector is paused, the data sync will be scheduled to be performed when the connector is re-enabled. Task definition author has the capability to configure the task to wait for success or failure, through Wait for Sync to Finish field.

Configuration examples


Image Added

User Scenario: Trigger Connector Re-sync

Image Removed



Action Output

Output TypeSuccessful ExecutionExample (Successful Execution)
EXTENSION

The extension output follows the standard Extension Output format, providing:

  • “exit_code“, “status“, “status_description“: General info regarding the task execution.

  • “invocation”

    • “fields”: The task configuration used for this task execution.

  • result fields

    • “latest_connector_details": Contains the latest received Connector Details. Information depends on the result set provided by Fivetran API. Consequently, the data contained in "latest_connector_details" can be backward compatible if Fivetran API retains backward compatibility.


Code Block
{
    "exit_code": 0,
    "status_description": "Task executed successfully.",
    "invocation": {
        "extension": "ue-fivetran",
        "version": "1.0.0",
        "fields": { ... },
    },
    "result": {
        "latest_connector_details": {
            "code": "Success",
            "data": {
                "destination_schema": {
                    "name": "github"
                },
                "id": "paving_diffuser",
                "group_id": "fistula_cheek",
                "connector_type_id": "github",
                "name": "github",
                "connected_by": "balsam_dilapidated",
                "created_at": "2023-05-23T12:51:13.378309Z",
                "succeeded_at": "2023-05-25T14:09:37.460Z",
                "failed_at": "2023-05-24T15:34:22.341Z",
                "paused": false,
                "pause_after_trial": false,
                "version": 0,
                "sync_frequency": 360,
                "schedule_type": "auto",
                "status": {
                    "setup_state": "connected",
                    "schema_status": "ready",
                    "sync_state": "scheduled",
                    "update_state": "on_schedule",
                    "is_historical_sync": false,
                    "tasks": [],
                    "warnings": []
                },
                "config": {
                    "sync_mode": "SPECIFIC_REPOSITORIES",
                    "repositories": [
                        "ue-fivetran/test-repo-ue-fivetran"
                    ],
                    "auth_mode": "PERSONAL_ACCESS_TOKEN",
                    "username": "fivetran.user@dev.com",
                    "pat": "******",
                    "use_webhooks": true
                }
            }
        }
    }
}



Failed ExecutionExample

  • “exit_code“, “status“, “status_description“: General info regarding the task execution.

  • “invocation”

    • “fields”: The task configuration used for this task execution.

  • result fields

    • “errors“: The errors reported by the extension task during the execution.

    • “execution_info”:

      • “url”: The endpoint that resulted in erroneous execution

      • “http_code”: The HTTP error from the error response if available


Code Block
{
    "exit_code": 1,
    "status_description": "Task execution failed.",
    "invocation": {


        "extension": "ue-fivetran",
        "version": "1.0.0",
        "fields": { ...
        },
    },
    "result": {
        "errors": [
            "Invalid authorization credentials"
        ],
        "execution_info": {
            "url": "https://api.fivetran.com/v1/groups",
            "http_code": 401
        }
    }
}

...



Input Fields

Field
Type
Default Value
Mandatory
Description
Introduced in Version
Action
Choice
Sync
Yes
The action performed upon the task execution.
Available options:
  • Sync
  • Re-sync (Historical Sync)
1.0.0
Credentials
Credentials

Yes

The API Key & Secret are used for Authentication. Credential definitions should be as follows:
  • As “Runtime User” the Fivetran API key
  • As “Runtime Password” the Fivetran API Secret
1.0.0
Destination Name
Dynamic Choice Field

Yes
The Destination / Group the Connector Belongs to.
1.0.0
Connector Name
Dynamic Choice Field

Yes
The Connector Name
1.0.0
Wait for Sync to Finish
Checkbox
False
No
Wait for the Connector Sync to be finished, either with Success or Failure
Fivetran does not provide specific APIs to retrieve the status of a specific Sync Instance, therefore identification of a Successful or Failed completion of a sync operation is implemented by means of a “Best Effort” algorithm.
1.0.0
Polling Interval (sec)
Integer
5
No
The polling interval in seconds on which the task is polling for the Fivetran Connector Sync status.
The minimum acceptable value is “1“. Default value is “5“.
Available only in case Wait for Sync to Finish = True
1.0.0
Force Sync
Boolean
False
No
If Force Sync is True and the connector is currently syncing, sync is stopped and restarted. If Force Sync is False, the connector will sync only if it isn't currently syncing.
Available only when Action = “Sync”
1.0.0
Output-Only fields

Field
Type
Description
Introduced in Version
Setup State
Text
The set-up state of the Connector. Retrieved/Updated before the Sync trigger and during polling.
1.0.0
Sync State
Text
The sync status of the Connector. Retrieved/Updated before the Sync trigger and during polling.
1.0.0
...
In all cases, if Wait for Sync to Finish = “True” the Task will wait until the Sync Status reaches Success or Failure.
...
Exit Codes
Extension Exit Code
Status
Status Description
 Meaning
0
SUCCESS
“Task executed successfully“
Successful Execution
1
FAIL
“Execution Failed: Task completed with errors. Refer to STDERR or Extension Output for more details.”
Generic error failure, when the error code does not match exit codes 3, 10, 20, 30, 31.
3
AUTHORIZATION_ERROR
“Authorization Error: Invalid or insufficient credentials.”
The given Fivetran API Key or Secret is unauthorized.
10
CONNECTION_ERROR
“Connection Error: Fivetran server responded with error.“
Fivetran server connection error.
20
DATA_VALIDATION_ERROR
“Data Validation Error: <error details>.“
Input fields Validation Error.
30
SYNC_FAILED
“Sync Failed: Sync or Re-sync was triggered, but completed with failure.“
Sync completed and identified as failed.
31
SYNC_MONITOR_FAIL
“Sync Monitor Failed: Sync or Re-sync was triggered, but an error occurred while monitoring for its completion.”
An error occurred during the monitoring phase. Monitoring is enabled by Wait for Sync to Finish, input field.
...
Modifications of this integration, applied by users or customers, before or after import, might affect the supportability of this integration. For more information refer to Integration Modifications.
Configure Universal Task
For a new Universal Task, create a new task, and enter the required input fields.
 Anchor
IntegrationModifications
IntegrationModifications
Integration Modifications
...
 Anchor
changelog
changelog
Changelog
ue-fivetran-1.0.1 (2023-10-18)
Fixes
  • Fixed: Execute the integration with a Python installation, other than the Agent-bundled one (#34545).
ue-fivetran-1.0.0(2023-06-08)
Initial Version