Github to Snowflake

The Github kit uses the Github API to connect and load the data into your data warehouse.

This Kit includes…

• 9 Action rivers to retrieve to the following entities:

  • Organization Teams

  • Organization Members

  • Repository Issues

  • Repository Commits

  • Repository Pull Requests

  • Repository Assignees

  • Repository Tags

  • Repository Collaborators

  • Repository Branches

• 9 Source to Target rivers to load the Github event data into Snowflake

  • For Organization level entities, the destination table format is: GITHUB_{github_owner}_{ENTITY}

  • For Repository level entities, the destination table format is: GITHUB_{github_owner}_{github_repo}_{ENTITY}

• 1 Logic River to orchestrate the ingestion and schedule it

Please note that the incremental load is only available for Repository Issues and Repository Commits, as only these 2 endpoints support a start date and end date in the query. For these 2 entities, the S2T river loads the data using Upsert-Merge method. For other entities, tables are overwritten.

Configuring this Kit for use

Before using this kit, you need to get your Github access token: https://developer.Github.com/how-to-authenticate-with-personal-access-tokens

Environment Variables

In this Kit, there are 4 environment variables that are used to make for dynamic use of target configuration. 

• {db_github} is used in all Target configurations and queries as the target database name.

• {schema_github} is used in all Target configurations and queries as the target schema name.

• {github_owner} is the github organization or owner of the repository, and this value is used to configure the target table name configuration

• {github_repo} is used to configure the target table name configuration

• {Github_Alert_Group} is used as the alert email address(es) for when the orchestration and source to target rivers fail (optional).

Go to the Variables menu on the left side navigation bar and create variables for db_github, schema_github, github_owner and github_repo. Fill in the values for the Snowflake Database/Schema you'd like the data to land.

Connections

Kits are imported with ‘blank’ source and target connections. In order to use the template, you have two options:

• Swap out blank connections with existing connections

• Add credentials to the blank connections that come with the imported template.

This kit requires a connection with the key "token" and its value

Have questions about this Kit or want more endpoints?

Send an email to helpme@rivery.io with the title “Help – Github to Snowflake Kit”