Pagination Options in Rest API

Introduction

Pagination is a common technique used to split large amounts of data into smaller, more manageable chunks. This technique is especially useful when dealing with large data sets that could slow down or crash the application if loaded all at once.
Rivery provides users with the ability to paginate data when working with large data sets in REST API.
This document will discuss how to implement pagination in Rivery using Action Rivers.

Implementation Procedure

To implement pagination in REST API, we will use the following steps:

Set up the Data Source

1. First, click on the “Create River” button and select Action River. You should see the following screen:
   

2. Select Rest Action

   

3. To establish your Custom API connection, follow to these guidelines.
   Test the connection, and refer to your API documentation for information on pagination.

Define the Pagination Parameters

1. Click on the “Results” tab and ensure that the response type is set to “Data”:
   

2. After clicking on "Pagination details", you can select the dropdown arrow to reveal 4 different options:
   

What do these 4 options mean?

No Pagination

The "No Pagination" option means that the API request will only provide the initial response without any additional pages.
However, if the response size is large, using this option could potentially decrease the speed of your River. It is a suitable choice only when the API call returns information that fits on a single page.

Paginate by Offset

Paginate by Offset - The simplest form of pagination is "Paginate by Offset". In the GET request, it can be represented as:

To specify the pages to return, "Paginate by Offset" uses the "Limit" and "Offset" variables in the GET call.
In the example mentioned above, the response will retrieve pages 100 to 120.
If you choose this option in Rivery, a page like the following will appear:

• “Offset Start” - If you want to fetch only pages 30-40 from a 100-page API request, you need to specify the "Offset Start" as 30. This value determines the starting point of the offset, which means that the API will return pages 30 to 40 based on the limit set in the GET request.

• "Offset Key" refers to the JSON data element that you wish to offset by. Usually, it is named "offset," but there are cases where it may be identified by another name such as "start_page" or "begin_page," depending on the nature of the response.

• “Page_Offset variable” - allows you to update the value of the next offset during each iteration by using a preconfigured variable named “Page_Offset”. To enable this feature, create a “Page_Offset” variable in the Variables tab, and check the “Update offset value into a page_offset variable in every page iteration” box:

  

  

• “Stop according to” is the criteria at which the Action River will stop paginating and thus pulling data from the API. This dropdown list gives you 2 options:

  • The "Page Size" option enables the user to determine the number of data rows included on a single page of the response and to stop the pagination once a page has less than the full number of entries (i.e., the last page). This option is particularly valuable when the number of pages is variable or unknown.
  • "Key in Response" option enables you to set a threshold for a specific key to restrict the number of pages returned.
    For instance, let's consider the example of fetching pages 30-40 from a 100-page API request. Here, you would specify the "Key in Response" as "page_number" and assign it a value of 30 to limit the number of pages returned.

• The "Sleep After Each Page" option is available universally, allowing you to set a time delay (in seconds) between each API pagination.

Paginate by Page

“Paginate by Page" allows you to iterate through all of the pages without an offset:

• The "Page Key" option lets you indicate the key in the JSON response that corresponds to the page number. This key might be named "page_number," "page_num," or simply "page".
• “Page_Value Variable” - allows you to update the value of the next page during each iteration by using a preconfigured variable named “page_value”. To enable this feature, create a “page_value” variable in the Variables tab, select the “Paginate By Page” option under Results → Data, and check the “Update the value of the next page into a page_value variable in every page iteration” box:
  

• “Stop according to” is the criteria at which the Action River will stop paginating and thus pulling data from the API. This dropdown list gives you 2 options:

  • The "Page Size" option enables the user to determine the number of data rows included on a single page of the response and to stop the pagination once a page has less than the full number of entries (i.e., the last page). This option is particularly valuable when the number of pages is variable or unknown.
  • "Key in Response" option enables you to set a threshold for a specific key to restrict the number of pages returned.
    For instance, let's consider the example of fetching pages 30-40 from a 100-page API request. Here, you would specify the "Key in Response" as "page_number" and assign it a value of 30 to limit the number of pages returned.

• The "Sleep After Each Page" option is available universally, allowing you to set a time delay (in seconds) between each API pagination.

Next Page In Response

This option is designed for situations where the pagination does not follow a standard cadence established by a variable in the GET call. Instead, it relies on information obtained from a previous GET response.
For instance, consider an example API that takes an "id" as an input parameter and returns customer information such as name, age, height, weight, income, and a value named "id_behind" that corresponds to the id of the person waiting behind you in the line.
To obtain the information for the next customer in line, you must make another API call by setting the id parameter to the "id_behind" value of the previous customer. Rivery provides 3 options to accomplish this task:

1. The "Static Url With Param From Response" feature lets you set a new URL for the next page of the response and map a variable from the previous response to a variable that you will use in the GET call for the next page of the response.
   • The "Next Page Key" refers to the key that holds the parameter you will use to retrieve the next page of the response. In the example mentioned earlier, the "Next Page Key" would be "id_behind".
   • “Send next page in key:” - This allows us to map the “Next Page Key” to the Key in the GET request for the next page. In the example above, this would be set to “id”.
   • The option "Send next page using a new url" enables you to specify a different URL for the next page's GET request. This can be useful if you want to retrieve information from a different database using a different REST URL and pass in the "id_behind" parameter from the previous GET request.

• “Stop according to” is the criteria at which the Action River will stop paginating and thus pulling data from the API. This dropdown list gives you 2 options:

  • The "Page Size" option enables the user to determine the number of data rows included on a single page of the response and to stop the pagination once a page has less than the full number of entries (i.e., the last page). This option is particularly valuable when the number of pages is variable or unknown.
  • "Key in Response" option enables you to set a threshold for a specific key to restrict the number of pages returned.
    For instance, let's consider the example of fetching pages 30-40 from a 100-page API request. Here, you would specify the "Key in Response" as "page_number" and assign it a value of 30 to limit the number of pages returned.

• The "Sleep After Each Page" option is available universally, allowing you to set a time delay (in seconds) between each API pagination.

2. The "Url In Response" option is used when the URL for the next page is stored within a key of the previous response.

• “Next page key” - Here, all we have to do is specify the key in the previous response that holds the full Url for the next page.

3. “Next Page in Body” - This option is used for when the information needed is stored in a two layer next page call (the ‘next to next page’) where the information for the next page (call this, page ‘b’) is stored in the previous response (call this, page ‘a’), but the page with the information we want to derive (call this, page ‘c’) can only be accessed via a key in page ‘b’.

• Page a → contains information to access → Page b → contains information to access-> Page c
• “Next page key” - This allows us to specify the key in the previous response (page ‘’a’) where the page we want to navigate to is located in (page ‘b’).
• “Send next page in key” - This allows us to specify the key in the page ‘b’ api call in order to access the page that we want (page ‘c’).

Dynamic Changing of Parameters

The parameters in many of these settings can be modified dynamically by using Variables, which can be created by clicking the button on the top right and identified using the {variable_name} notation.
By inserting these variables into the parameter boxes, you can alter the pagination of the API call based on the requirements of your Logic River workflow.