Pagination

Blueprint Pagination Configuration Guide

Overview

This guide explains how to configure pagination in Rivery Blueprints, including the new break conditions available for controlling when data retrieval should stop.

What is Pagination in Blueprints?

When working with APIs that return large datasets, pagination allows the retrieval of data in manageable chunks rather than fetching everything at once. Blueprints support several pagination styles and configurable break conditions to determine when to stop retrieving data.

Supported Pagination Types

Blueprints support the following pagination types:

• Offset-based Pagination
  Uses an offset parameter to define the starting record for the request.

• Page-based Pagination
  Uses a page parameter to indicate which page of results to retrieve.

• Cursor-based Pagination
  Uses a token or cursor to point to the next set of results.

Steps to Configure Pagination

This section provides step-by-step instructions for configuring pagination in a Blueprint.

1. Define Pagination Type

Select the pagination type that matches your API requirements:

• offset
• page
• cursor

2. Set Pagination Parameters

Specify parameters required for the chosen pagination type.

Offset Pagination Example

pagination:
  type: offset
  parameters:
    - name: offset
      value: 0
      increment_by: 10
  location: qs

Cursor Pagination Example

pagination:
  type: cursor
  parameters:
    - name: cursor
      value: ""
      token_location: response
      token_format: json
      token_path: "$.next_cursor"
  location: qs

3. Configure Break Conditions

Break conditions determine when pagination should stop. Rivery Blueprints support several advanced break conditions.

Page Size Break Condition

Stops pagination when the number of items returned in a page is fewer than the page size.
Example Use Case: An API returns fewer items than the requested page size on the final page.

pagination:
  break_conditions:
    - name: BreakIfPageSmallerThanPageSize
      condition:
        type: page_size_break
        page_size_param_name: per_page
        items_json_path: "$.data"

Total Items Break Condition

Stops pagination when the total number of items processed equals or exceeds the total count provided in the API response.
Example Use Case: The API includes a field indicating the total number of available records.

pagination:
  break_conditions:
    - name: BreakIfAllItemsReceived
      condition:
        type: total_items_reached
        page_size_param_name: per_page
        page_param_name: page
        total_items_path: "$.total_count"

Empty JSON Path Break Condition

Stops pagination if a specified JSONPath in the response is empty, null, or missing.
Example Use Case: An API signals the end of data by returning an empty array or removing a property.

pagination:
  break_conditions:
    - name: BreakIfEmptyJsonPath
      condition:
        type: empty_json_path
        key_json_path: "$.data"

Boolean Break Condition

Stops pagination when a specific boolean field in the response indicates no additional pages are available.
Example Use Case: An API includes a flag such as has_more or isLastPage.

pagination:
  break_conditions:
    - name: BreakIfHasMoreIsFalse
      condition:
        type: boolean_break
        key_json_path: "$.has_more"
        field_break_value: false

4. Choose Parameter Location

Define where pagination parameters should be included in the request:

qs → Query string

url → URL path

body → HTTP request body

5. Validate the Configuration

Before running a Blueprint with pagination:

• Confirm all parameter names match the API specification.

• Verify that token paths are accurate.

• Always test break conditions to avoid infinite loops.

Full YAML Example

Below is a full YAML example demonstrating page-based pagination with the Page Size Break Condition:

connector:
  name: ExampleAPI
  type: rest
  base_url: 'https://api.example.com'
  default_headers:
    Authorization: 'Bearer {ACCESS_TOKEN}'
    Content-Type: "application/json"

steps:
  - name: GetItems
    description: Retrieve items with page-based pagination
    type: rest
    endpoint: "{{%BASE_URL%}}/items"
    http_method: GET
    pagination:
      type: page
      parameters:
        - name: page
          value: 1
          increment_by: 1
        - name: per_page
          value: 100
      break_conditions:
        - name: BreakIfPageSmallerThanPageSize
          condition:
            type: page_size_break
            page_size_param_name: per_page
            items_json_path: "$.data"
      location: qs
      variables_output:
        - variable_name: items_output
          response_location: data
          variable_format: json
          transformation_layers:
            - type: extract_json
              json_path: "$.data"

Summary of Pagination Break Conditions

Best Practices

• Always define a break condition to avoid infinite pagination loops.
• Choose the simplest break condition your API supports.
• Test your configuration with both small and large datasets.
• Confirm all JSONPath references match your API’s actual response structure.