Google Cloud Storage Connection

Introduction

This guide provides step-by-step instructions on how to set your Google Cloud credential JSON key, create a Google Cloud Storage bucket, and obtain the necessary credentials for using Google Cloud Storage with Rivery. By the end of this guide, you will have configured Rivery to connect with your Google Cloud Storage.

Prerequisites

Ensure you have signed up for Google Cloud Platform and have an admin user for the Google Cloud Console. If not, you need to sign up here.
If you don’t have one of these prerequisites, you can start here .

Create a Service Account User for Rivery

Rivery uses Google Cloud Storage to upload your source data. Follow these steps to create a user in the Google Cloud Platform Console with access to the relevant bucket and BigQuery Project.

Steps to Create a Service Account

1. Sign into Google Cloud Platform Console.

2. Go to IAM & Admin

3. Click on the Create Service Account button.

4. In the Service Account Wizard, set your Service Account name (e.g., RiveryUser) and click Create and Continue.

5. Create a Key for the Service Account:

• Assign the BigQuery Admin role.
• Click on Add Another Role and assign the Storage Admin role.
• Copy your Service Account Email for later use in Rivery.

6. Now lets create a key for the service account:

   1. Go to the service account screen, locate the serivce you've just created and click on it

   2. In the new service account screen cick on Key

   3. Click on add key
      

   4. Choose key type JSON and click on create

   5. Your JSON secret key will be download. keep it in a safe place.
      

Enable Cloud Storage and GCS API

1. Go to  API's & Services ->Click on ENABLE APIS AND SERVICE button.

2. Search for  Google Cloud Storage JSON API and click on Enable API .

Create a Google Cloud Storage Bucket

Rivery needs a Google Cloud Storage bucket to be a FileZone before your data is loading up to BigQuery. You can either use the FileZone bucket or objects as a base to other Hadoop or Apache Spark operation by Google Data PROC, or by your other services.

So, let's create a Google Cloud Storage bucket for Rivery:

1. Sign into Google Cloud Platform Console.

2. Go to Storage -> Browse - > click on CREATE BUCKET button.

3. In the wizard:

   1. Set Bucket Name example : project_name_rivery_file_zone

   2. Set your Bucket to be Regional (Multi-Region is not stable for loading)

      and choose your preferred location

   3. Click CREATE button.

Configure your Google Cloud Storage Bucket in Rivery

Let’s create a new connection for your Google Cloud Storage

1. Go to Connections.

2. Press on New Connection .

3. From the source list, choose  Google Cloud Storage

4. Now, enter your credentials information for Google Platform Service Account

1. Connection Name

2. Description (optional)

3. Project Id  (can be found on Google Platform Home section)

4. Project Number (optional - can be found on Google Platform Home section)

5. Service Account email  - it's Service Account Id that you used to copy the Service Account Wizard.

6. Choose file - the JSON credentials file that was generated at the end of Service Account Wizard.

7. Region - the region your bucket was created at

8. Default bucket - the defualt bucket Rivery will use(the one you've created)

Click 'Test Connection' at the bottom to test! Once a valid connection is made, save the connection.

If you cannot get a valid connection set up, contact helpme@rivery.io for support.

Known Issues

• Sometimes the "Storage Admin" type user role does not have a certain permission "storage.buckets.get" given to it by default:

In this case, you will have to edit your GCP user roles by duplicating that "Storage Admin" role by clicking

, making sure the custom role you create has the storage.buckets.get permission, then assigning your service account this custom role instead of the "Storage Admin" (See the "Create a Service Account User for Rivery" section of this document).

Conclusion

This guide showed you how to create a Service Account user for Rivery and Cloud Storage Bucket

You now have a Google Cloud Storage connection that you can use in every river that targets to it and also as a source.