Configure Custom FileZone for Firebolt
  • 3 Minutes to read
  • Dark
    Light
  • PDF

Configure Custom FileZone for Firebolt

  • Dark
    Light
  • PDF

Creating a custom FileZone in Rivery gives you the opportunity to manage your data in your own S3 service, in your own AWS. Therefore, your data can be saved for more than 24 hours in your S3 bucket. You can either use the FileZone bucket or objects as a base to other Hadoop or Spark operations by Amazon EMR, or by your other services.

Before you use this guide, please make sure you’ve signed up for AWS and you have a console admin user. If you don’t have one of these prerequisites, you can start here.

Create an S3 Bucket

Firebolt needs the data to be loaded to an S3 bucket to serve as the basis for an external table for further ingestion.

Note: You can find the up to date documentation of S3 operations and getting started here .

So, let's create an S3 bucket for Rivery:

  1. Go to S3 Management in AWS Console .

  2. Click on Create Bucket

Give the bucket a name, and choose the same region your firebolt database will be on. Use the S3 wizard defaults by reviewing and following the wizard screen, and click on Create Bucket .
configure-custom-filezone-for-snowflake_mceclip92.png

Create an AWS User for Rivery

Rivery uses Amazon S3 bucket to upload your source data into it and then pull that data to your target. Therefore, there is a need to create a user in Amazon Platform Console, that will have the relevant bucket.

So, first of all - let’s create a user for Rivery and we'll give it permission to manage and read the FileZone bucket.

How do we do that?

  1. Sign into AWS Console

  2. Go to IAM in AWS Console -> Users - > Add User

  3. In the wizard, set your username (i.e: RiveryUser), and check the Access type to Programmatic access. Click Next .

    configure-custom-filezone-for-snowflake_image42.png

  4. In the Set Permissions screen, choose Attach Policies Directly.

  5. In order to create new permission for the user, click on Attach Existing Policy Directly and then on Create Policy. A new tab with AWS policy generator should be opened.

    configure-custom-filezone-for-snowflake_mceclip114.png

  6. In the policy generator screen opened, click on JSON tab.

    configure-custom-filezone-for-snowflake_mceclip25.png

  7. Paste the permissions configuration below to the blank text box that opens and press Review Policy. Please don’t forget to replace the bold areas with your current settings, as you set before.
    Replace the  <RiveryFileZoneBucket> with the S3 bucket name you've just created.

  {
 "Version":"2012-10-17",
 "Statement":[
   {
    "Sid":"RiveryManageFZBucket",
    "Effect":"Allow",
    "Action":[
    "s3:GetBucketCORS",
    "s3:ListBucket",
    "s3:GetBucketLocation"
     ],
    "Resource":"arn:aws:s3:::<RiveryFileZoneBucket>"
   },
   {
    "Sid":"RiveryManageFZObjects",
    "Effect":"Allow",
    "Action":[
      "s3:ReplicateObject",
      "s3:PutObject",
      "s3:GetObjectAcl",
      "s3:GetObject",
      "s3:ListObjects*",
      "s3:PutObjectVersionAcl",
      "s3:PutObjectAcl",
      "s3:ListMultipartUploadParts"],
    "Resource":"arn:aws:s3:::<RiveryFileZoneBucket>/*"
  },
  {
     "Sid":"RiveryHeadBucketsAndGetLists",
     "Effect":"Allow",
     "Action":"s3:ListAllMyBuckets",
     "Resource":"*"
  }
 ]
}
  1. Name your policy (i.e. RiveryFileZoneS3Policy ) and click on Create Policy.

  2. Go back to your browser tab of the user creation (it should be opened already), Refresh the policies list using the button in the right, search and Mark the policy just created, and click on Next: Tags.

  3. End the process by clicking on Next: Review and then on Create User.

  4. In the Complete section, you will get the user’s AWS credentials (aws_access_key_id, aws_secret_access_key) and the user's ARN .
    Save the credentials in a safe place!
    You will need all of them afterward, in Rivery . configure-custom-filezone-for-snowflake_mceclip82.png

  5. Now, the User should have permissions to manage and read the FZ bucket created for Rivery. Please make sure the policy you've created attached to the user created.
    configure-custom-filezone-for-snowflake_mceclip64.png

Configure custom FileZone for Firebolt in Rivery

Now, let's configure the bucket we've just created as a Custom FileZone in Rivery.

  1. Open your Firebolt Connection, by going to Connections->Create New Connection, and choose Firebolt.

  2. In the connection, check the Custom File Zoneconfigure-custom-filezone-for-snowflake_mceclip011.png

  3. If you have already an S3 connection, you can set it here as a Custom FileZone connection or you can create a new connection.

  4. In the form, set your Aws Access Key and Aws Access Secret Key, and Region Key .

  5. Name your S3 File Zone Connection and Save.

  6. Now you can test your connection .

  7. After saving, Choose your default bucket for your FileZone area. Use the bucket you've created in above:

    configure-custom-filezone-for-snowflake_mceclip2_1.png

  8. Save the Fireblot connection.


Was this article helpful?