Data Connector Example
How to use the Data Connector capsules using Snowflake as an example
Data Connectors are a convenient way to bring in data from external data lakes into Code Ocean. Each data connector consists of two capsules, one which performs a query and outputs a data file (in parquet, text or other tabular format) and one which will create a data asset automatically from the results. After duplication, a user can simply use the data asset generation capsule and create data assets from SQL queries without any extra steps.
Data Connector Setup
Duplicate the Snowflake - Data Connector and Snowflake - Data Asset Generation capsules into your VPC environment.
In the Snowflake - Data Connector capsule add your Database Credentials and Account to the capsule as secrets. See README file for information on how to find the Account in Snowflake.
AWS Data Connectors can use Assume Roles rather than user secrets to establish credentials
Run a test query on the Snowflake - Data Connector from the App Builder. Follow the README for instructions on how to get the Warehouse, Database, Schema, etc.
Debug any connection issues until you are able to run a successful query.
Go to the Metadata tab and save the capsule id
Go to the Snowflake - Data Asset Generation capsule.
In the code/config.sh file edit “snowflake_query” to match the metadata for the Snowflake - Data Connector capsule. Also, be sure to edit the “co_domain” to match your Code Ocean deployment. If you expect your queries to take over an hour, adjust the “max_execution_time” accordingly.
Run a test query on Snowflake - Data Asset Generation from the App Builder using the same connection information as in the Snowflake - Data Connector capsule. Edit the Data Asset parameters as needed.
Check your data assets for the results of your query!
Last updated
