Setting Up Google Cloud Storage as a Target
  • 22 Jun 2020
  • 3 Minutes To Read
  • Contributors
  • Print
  • Share
  • Dark
    Light

Setting Up Google Cloud Storage as a Target

  • Print
  • Share
  • Dark
    Light

Setting Up Google Cloud Storage (GCS) As Your Target Data Platform

Overview

Welcome to Getting Started with Rivery and Google Cloud Storage (GCS).

This guide will show you how to set your Google Cloud p12 key, how to create a Google Cloud Storage bucket, and how to get the right credentials for using Google Cloud Storage with Rivery. In the end of the guide, you will make a quick setup in Rivery to connect your Google Cloud Storage.

Before you use this guide, please make sure you’ve signed up for Google Platform and you have a console admin user.

If you don’t have one of these prerequisites, you can start here .

Create a Service Account User for Rivery:

Rivery uses Google Cloud Storage bucket to upload your source data into it. Therefore, there is need to create a user in Google Cloud Platform Console, that will have the access to the relevant bucket and to the relevant BigQuery Project.

So, first of all - let’s create a user for Rivery.

How do we do that?

  1. Sign into Google Cloud Platform Console.

  2. Go to IAM & Admin  -> Service account - > click on CREATE SERVICE ACCOUNT button.

  3. In the wizard
    service_account_wizard_v2.png

    1. Set your Service Account name (i.e: RiveryUser) and

    2. Set Roles :

      • Storage Admin
    3. Copy your Account Service ID / Email from the service account list. 
      Later, you will use this to enter it in a Rivery connection.

    4. Mark the Checkbox - Furnish a new private key and in the key type options :
      choose P12.

    5. Click on CREATE button

      Your P12 secret key will be download. keep it in a safe place.

Enable Cloud Storage and GCS API

  1. Go to  API's & Services ->Click on ENABLE APIS AND SERVICE button.

  2. Search for  Google Cloud Storage JSON API and click on Enable API .

Create a Google Cloud Storage Bucket

Rivery needs a Google Cloud Storage bucket to be a FileZone before your data is loading up to BigQuery. You can either use the FileZone bucket or objects as a base to other Hadoop or Apache Spark operation by Google Data PROC, or by your other services.

So, let's create a Google Cloud Storage bucket for Rivery:

  1. Sign into Google Cloud Platform Console.

  2. Go to Storage -> Browse - > click on CREATE BUCKET button.

  3. In the wizard:

    1. Set Bucket Name example : project_name_rivery_file_zone

    2. Set your Bucket to be Regional (Multi-Region is not stable for loading)

      and choose your preferred location

    3. Click CREATE button.

Configure your Google Cloud Storage Bucket in Rivery

We need to make a quick setup in Rivery in order to start using it with the Google Cloud Storage Bucket.

  1. Log into Rivery.

  2. Now we’ll set your bucket as the default Google Cloud Storage bucket in Rivery:

    1. In the main menu, go to Variables

    2. Set your {GCS_file_zone} variable value to the s3 bucket name that you’ve created. That will be saved automatically. You don’t need to press save anywhere.

    3. If you don’t have the {GCS _file_zone } variable, you should add a new variable with that name and the s3 bucket name that you’ve created as value. Press on + Add VariableScreen_Shot_2017-11-26_at_6.56.36_PM1.png

  3. Let’s create a new connection for your Google Cloud Storage

    1. Go to Connections.

    2. Press on New Connection .

    3. From the source list, choose  Google Cloud StorageScreen_Shot_2017-11-26_at_7.05.29_PM.png

  4. Now, enter your credentials information for Google Platform Service Account

    Screen_Shot_2017-11-26_at_7.08.59_PM.png

    1. Connection Name

    2. Description (optional)

    3. Project Id  (can be found on Google Platform Home section)

    4. Project Number (can be found on Google Platform Home section)

    5. Service Account email  - it's Service Account Id that you used to copy the Service Account Wizard.

    6. Choose file - the file P12 file that was generated at the end of Service Account Wizard.

Conclusion

This guide showed you how to create a Service Account user for Rivery and Cloud Storage Bucket

You now have a Google Cloud Storage connection that you can use in every river that targets to it and also as a source.

Was This Article Helpful?