Custom FileZone for Snowflake

This is a step-by-step tutorial to setting up a Custom FileZone for Snowflake.

Overview

Setting up a Custom File Zone in Rivery is an optional feature, with the default option relying on the Managed File Zone provided by the platform, which requires no setup.
The main advantage of setting up a Custom File Zone is the ability for organizations to ensure that data is stored within their own file zones, as opposed to being stored in Rivery's Managed File Zone. This also enables organizations to use the Custom File Zone as a data lake, where raw data can be stored before it is loaded into a Target cloud data warehouse. Furthermore, organizations have the ability to define their own retention policies for data stored in the Custom File Zone. Rivery's Managed File Zone (default) retains data for a period of 48 hours.

Rivery allows you to manage your data in your own S3 or Azure service. Your data will be stored in your bucket. A bucket is a container that holds objects. An object is a file and any metadata that describes that file.

Prerequisites

• If you're new to Amazon S3 buckets, start with the S3 documentation.

• If you're new to Azure Blob Storage, start with the Microsoft documentation.

Amazon S3 Buckets

Create Your S3 Bucket

1. Sign in to the AWS Management Console and open the Amazon S3 console

2. Click the Create Bucket button.

3. Fill in a Bucket name.
   The bucket's name must include:

• Be unique across all of Amazon S3.
• The length should be between 3 and 63 characters.
• Uppercase characters are not allowed.
• Begin with a lowercase letter or a numerical value.

Note:
You can't change the bucket's name once it's been created. See Bucket Naming Rules for more information on bucket naming.

4. Choose an AWS Region , the same region where your Redshift resides.

Note:
To reduce latency and costs while meeting regulatory requirements, choose a Region close to you.

5. Select Next.
   

6. Choose your preferred settings by following the on-screen instructions on the console.

7. Click Create Bucket.

Create an AWS User for Rivery

Create a user for Rivery and grant it permission to manage and read the FileZone bucket by following these steps:

1. Sign in to the AWS Management Console and open the Amazon S3 console.

2. Navigate to Users - > Add User.

3. In the console, set your username, and check the Access type to Programmatic access.

4. Click Next: Permissions.

5. In the Set Permissions form select Attach Existing Policies Directly.

6. Click Create Policy.

7. Navigate to the JSON tab.

8. Copy and paste the following policy:
   Note:
   Replace <RiveryFileZoneBucket> with the name of your S3 bucket.

{
 "Version":"2012-10-17",
 "Statement":[
   {
    "Sid":"RiveryManageFZBucket",
    "Effect":"Allow",
    "Action":[
    "s3:GetBucketCORS",
    "s3:ListBucket",
    "s3:GetBucketLocation"
     ],
    "Resource":"arn:aws:s3:::<RiveryFileZoneBucket>"
   },
   {
    "Sid":"RiveryManageFZObjects",
    "Effect":"Allow",
    "Action":[
      "s3:ReplicateObject",
      "s3:PutObject",
      "s3:GetObjectAcl",
      "s3:GetObject",
      "s3:ListObjects*",
      "s3:PutObjectVersionAcl",
      "s3:PutObjectAcl",
      "s3:ListMultipartUploadParts"],
    "Resource":"arn:aws:s3:::<RiveryFileZoneBucket>/*"
  },
  {
     "Sid":"RiveryHeadBucketsAndGetLists",
     "Effect":"Allow",
     "Action":"s3:ListAllMyBuckets",
     "Resource":"*"
  }
 ]
}

9. Select Review Policy.
10. Give the Policy a name and click Create Policy.

11. Using the rounded arrows on the upper right, refresh the list of policies, check the policy you just created, and select Next: Tags.

12. Click Next: Tags, Next: Review and then Create User to complete the process.

13. In the summary screen, you'll find the user's AWS credentials (Access key id and Secret access key), which you can download as a CSV file (this is the only time you'll be able to do so).

14. The User should now be able to manage and read the FileZone bucket that was created for Rivery. Please double-check that the policy you created is linked to the user you created.

Connect to Amazon S3

To connect to Amazon S3, see our Amazon S3 Connection documentation.

Azure Blob Storage Container

Please refer to our Azure Blob Storage Connection documentation to create a Standard Azure account and container.
Please Note: For Azure Blob Storage containers to work properly with Snowflake, a SAS token must be entered in the connection form.
To create a SAS Token, consult the Microsoft documentation.

Once you've finished creating the container and connecting to Azure Blob storage in Rivery, continue to 'Configure Custom FileZone for Snowflake in Rivery' down below.

Configure Custom FileZone for Snowflake in Rivery

1. Go to Connections -> +New Connection and search for Snowflake.

2. Type in your Connection Details and Credentials.
   

3. Toggle the Custom File Zone to true.

4. Pick your cloud storage option.

5. By clicking on FileZone Connection, you can select the previously configured FileZone Connection.

6. Choose a Default Bucket (Container) from the drop-down list.

7. Use the Test Connection function to see if your connection is up to the task.
   If the connection was successful, click Save.