- 3 Minutes to read
Connect to Any REST API Using Action Rivers
- 3 Minutes to read
This guide explains how to use Rivery's Action River capability to connect to any REST endpoint and use it as a new data source. Even though Rivery regularly adds native support for various data sources, there may be instances where the data source you need to connect to is not available on Rivery's list of supported sources. In such cases, this guide can be useful and help you avoid unnecessary coding.
This guide will demonstrate the process of creating a connection to an API endpoint using the GET method to retrieve a list of users. Each step of the process is covered and explained in a video that is provided at the end of the guide.
To create a connection to an API, the user needs to provide the API URL. If the user is creating a Source, they will make a GET call to the API.
A username and password, OAuth2, or Multi-Bearer authentication can all be used for authentication.
This API endpoint is a public endpoint, so authentication is not required.
Testing the Connection
In this example, the user has access to an API endpoint that returns a list of users. By selecting the "Test REST Action" button, the user can verify the connection. This action provides a success code of 200 (connection succeeded) and a sample dataset.
Request and Results Menu
When working with a REST API, these common options found in the Request and Results menus may come in handy:
- Adding Parameters
- Pagination Methods
- Sending a request in a loop.
Additional parameters, such as a last modified date, can be added to filter the retrieved data. For example, specifying a last modified date of one day will retrieve only the users who were added or modified within the past day.
When the "Add" button is clicked under "REST URL Params," the specified parameter will automatically be appended to the URL, eliminating the need for manual inclusion.
This process is not visible to the user, but behind the scenes, the parameter is added to the URL in the following format:
To handle large data sets, Rivery provides pagination methods that allow users to retrieve and process data in smaller chunks.
There are several pagination methods available in Rivery, including:
- Pagination by Offset
- Pagination by Page
- Next Page in Response.
The specific method used will depend on the data source and the requirements of the integration or transformation task. For further details, see our Pagination -Community article.
Sending a Request in a Loop
Rivery enables to set up a loop that repeatedly runs a request until a certain stop value is reached. This can be useful for situations where data is being constantly updated and needs to be periodically retrieved and processed.
Creating a Custom Connector
Once a connection to a REST endpoint has been established, it can be saved as a custom connector using the REST API Source. To do this, create a new Source to Target River, select the REST API Source you just created, and then proceed to configure your Target in the same way you would for any other Source to Target River.
Automating Table Structure Mapping
In this case, the user chooses a Source to Target River and select the REST API as the Source and Snowflake as the Target. Now, Rivery handles the task of using the JSON payload from the API endpoint to define the target schema. The user can easily use the "Auto Mapping" button and Rivery will automatically map the table structure and data types based on the metadata retrieved from a sample test.
After the table structure has been generated, data can be ingested by running the River. Data will be retrieved from the API based on the specified parameters (such as last modified date) and pushed into the target (in this case, Snowflake) using the specified primary key for merging.
Here is a video guide for this particular use case: