...
Your use of this download is governed by Stonebranch’s Terms of Use, which are available at https://www.stonebranch.com/integration-hub/Terms-and-Privacy/Terms-of-Use/
...
Overview
This Universal Task allows Stonebranch users to orchestrate, schedule, trigger, and monitor the Snowflake load and unload process from different data sources (cloud storage or local VM’s) directly from Universal Controller. It uses Python libraries to perform all functions listed in the following sections. Alternatively, you also can perform all these operations using the snowflake JDBC driver which you can add to the Universal Controller libraries, and use SQL Task to perform any operations with Snowflake (https://docs.snowflake.com/en/user-guide/jdbc-download.html)
...
Users can orchestrate the Snowflake functionalities using the following features available in the Universal Task
...
Software Requirements for Universal Template and Universal Task
Requires Python 3.6 or higher. Tested with the Universal Agent bundled Python distribution.
Python modules required:
- https://raw.githubusercontent.com/snowflakedb/snowflake-connector-python/v2.3.9/tested_requirements/requirements_36.reqs
- snowflake-connector-python
- snowflake-ingest
- Please refer to Snowflake URL : https://docs.snowflake.com/en/user-guide/python-connector-install.html for the latest Python connector details.
...
Universal Agent for Windows x64 Version 67.6 0 and later with Python options installed.
Universal Agent for Linux Version 67.6 0 and later with Python options installed.
...
Universal Controller Version 67.60.0.0 and later.
Software Requirements for the Application to be Scheduled
...
To use this downloadable Universal Template, you first must perform the following steps:
- This Universal Task requires the Resolvable Credentials feature. Check that the Resolvable Credentials Permitted system property has been set to true.
- Copy or Transfer To import the Universal Template file to a directory that can be accessed by the Universal Controller Tomcat user.
In the Universal Controller UI, select Configuration > Universal Templates to display the current list of Universal Templates.
Right-click any column header on the list to display an Action menu.
Select Import from the menu, enter the directory containing the Universal Template file(s) that you want to import, and click OK.
- into your Controller, follow the instructions here.
- When the files have been imported successfully, refresh the Universal Templates list; the Universal Template will appear on the list.
Configure Snowflake Integration Universal Task
...
Field | Description |
Snowflake Account | Provide the Snowflake user account; for example, sr14548.eu-central-1. |
Snowflake Host URL | Specify your host information in the form of a URL; for example, sr14548.eu-central-1.snowflakecomputing.com. |
Snowflake Login Name | Specify your Snowflake login name.. |
Warehouse | Specify the warehouse name; If not specified, connects to the default warehouse for the user. |
Log Level | Select a log level. |
Select a Snowflake function | Select the required Snowflake function (if the required function not available, check the SQL task to invoke snowflake DB). |
Snowflake Command | Provide a Snowflake command either in SQL or put / get commands etc. |
Local file Name & Path | Provide the local file name that need to be copied to stage. |
Stage Name | Provide the stage(internal) name in Snowflake. |
Encrypted Private Key | Select only if you have a encrypted private key. |
Stage File Name(s) | Provide the stage file names; if there are multiple files, separate by comma. |
Pipe Name | Specify the fully-qualified name of the pipe to use to load the data. |
Private Key File Path | Provide the private key file path to establish connection to Snowflake for data ingest. |
Private Key Password | Provide the password for private if it is encrypted. |
Snowflake Table Name | Provide the full path and the Snowflake table name where the table to be loaded. |
Use AWS Credentials | Check this if you need to supply AWS access key credentials. |
AWS Key ID & Secret Access Key | Provide the AWS secret Access Key (runtime user AWS Key id and secret key in the password section). |
AWS Storage Integration | Provide the name of the AWS storage integration created in Snowflake. |
Load using Pattern | If you need load the data using pattern, check this option. |
S3 Bucket URL | Provide your S3 bucket URL s3://<your_s3_bucket>/data/. |
Azure container File | Provide the Azure container file URL starting with azure:// |
Azure Storage Integration | Provide the Azure storage integration created in Snowflake. |
GCP storage URL | Provide the bucket name and the file name; for example, gcs://mybucket/data/files. |
Azure Sas Token | Provide the Azure Sas Token. |
Existing File Format Name | Specify an existing named file format to use for loading data into the table. |
Storage Integration | Provide the Snowflake storage integration details. |
Pattern | Regular expression pattern string - specifying the file names and/or paths to match. |
File Format Name | Specifies an existing named file format to use for loading data into the table. |
Format Type | Specify the format type: CSV | JSON | AVRO | ORC | PARQUET | XML. |
File Format Type Options | Specify the file format type options; for example, FIELD_DELIMITER = '|' if CSV type -- Refer to https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html |
Copy Options | Include other copy options; for example, ON_ERROR = CONTINUE or FORCE=TRUE- Refer to https://docs.snowflake.com/en/sql-reference/sql/copy-into-table.html |
Polling Interval (Secs) | Provide the Polling Interval time in Seconds, in the case of data ingestion from internal staging to Snowflake Table. |
Number of times to Poll | Specify the number of times to poll; otherwise, default is set to 50, in the case of data ingestion from internal staging to Snowflake Table. |
...
This document references the following documents:
Name | Location | Description | Universal Templates|
---|---|---|---|
https://docs.stonebranch.com/confluence/display/UC70/Universal + Templates | User documentation for creating Universal Templates in the Universal Controller user interface. | ||
Universal Taskshttps://docs.stonebranch.com/confluence/display/UC70/Universal+ Tasks | User documentation for creating Universal Tasks in the Universal Controller user interface. |
...