Pagination

Creating a Blueprint File with Pagination

This guide explains how to define pagination settings in a YAML format. Follow these steps to structure your pagination logic correctly in your Blueprint file.

1. Pagination Main Model

The main Pagination object specifies the type of pagination used and its settings. Here’s a breakdown of each field:

Pagination:
  type: "offset"  # Type of pagination (e.g., offset, page, cursor)
  parameters:
    - name: "offset"
      value: 0
      increment_by: 5  # Increment logic for pagination
  break_conditions:
    - name: "BreakIfHasMoreIsFalse"
      condition:
        type: "string_equal"
        value: "false"
        variable: "{{%has_more%}}"  # Condition to break the loop
  location: "qs"  # Location where pagination data should be placed (query, body, etc.)

Explanation:

• type: Defines the type of pagination (offset, page, or cursor).
• parameters: A list of pagination parameters. Each parameter contains:
  • name: The dynamic name (e.g., offset, limit).
  • value: The default starting value.
  • increment_by: How much the parameter increases with each pagination step.
• break_conditions: Conditions that stop pagination (e.g., when a specific variable has a certain value).
• location: Specifies where pagination data should be placed (e.g., query string, body).

2. Increment Parameter for Pagination

Use PaginationIncrementParameter for incremental pagination based on offset or page size. Here’s how to define it in YAML:

PaginationIncrementParameter:
  type: "offset"
  name: "offset"  # Parameter name for pagination
  value: 0  # Default value for pagination parameter
  increment_by: 5  # Step size for incrementing the parameter

Explanation:

• type: Can be either offset or page.
• name: The name of the pagination parameter (e.g., offset, limit).
• value: The default starting value.
• increment_by: How much to increment the value on each pagination cycle.

3. Cursor Parameter for Pagination

Use PaginationCursorParameter for cursor-based pagination. Here’s the YAML format:

PaginationCursorParameter:
  type: "cursor"
  name: "cursor"  # Parameter name for cursor-based pagination
  value: "https://example.com/api/v1/?since_id=12345"  # Default cursor value
  token_location: "response"  # Where to extract the pagination data from
  token_format: "json"  # Format of the token (e.g., JSON, RFC8288)
  token_path: "$.token"  # Path to extract pagination data

Explanation:

• type: Specifies cursor-based pagination.
• name: The cursor name (e.g., cursor, token).
• value: Default starting point of the cursor.
• token_location: Where to find the token (response or header).
• token_format: Specifies the token format (JSON, RFC8288).
• token_path: JSONPath or header path to extract the pagination token.

4. Enums

To simplify handling of various fields, the following enums are used:

PaginationTypeEnum:
  - offset
  - page
  - cursor

PaginationDataLocationEnum:
  - qs
  - url
  - body

TokenTypeEnum:
  - response
  - header

TokenFormatEnum:
  - RFC8288
  - json

5. Break Conditions

# Break Conditions for Pagination Logic in Rivery

These conditions allow you to stop the pagination process based on specific criteria, ensuring your data extraction is efficient and stops when the desired data is retrieved.

Current Supported Break Conditions

We currently support the following break conditions in Rivery's pagination logic:

• NumberOperatorCondition
• JsonItemResultsCount
• StringCondition
• OrCondition

These conditions allow users to handle different scenarios while fetching data using pagination in REST API calls.

Enriching Break Conditions for Pagination Logic

To enhance the control over pagination in various use cases, we propose adding additional pagination-specific break conditions. Each condition has a structure that includes:

• Type: The condition type to be applied.
• Value: The value or criteria that should trigger the condition.

These conditions enable flexible pagination logic, ensuring that the loop stops based on specific, real-world scenarios commonly encountered when dealing with APIs.

Breaking Conditions - Examples

1. FieldNotExistsCondition

Description:

This condition stops pagination when a "next" token does not exist, i.e., when the key or field is missing from the API response.

YAML Example:

break_conditions:
  - name: BreakIfNextPageTokenExists
    condition:
      type: field_not_exist
      value: true
    variable: "{{%next_page_token%}}"

Code Example:

break_condition = VariableBreakCondition(
    name="BreakIfNextPageTokenExists",
    condition=FieldNotExistsCondition(value=False),
    variable="{{%next_page_token%}}"
)

2. BooleanEqualCondition

Description:

This condition stops pagination based on a boolean flag (e.g., true or false). It is commonly used when checking if a flag like has_more is set to a specific value.

YAML Example:

break_conditions:
  - name: BreakIfHasMoreIsFalse
    condition:
      type: boolean_equal
      value: false
    variable: "{{%has_more%}}"

Code Example:

break_condition = VariableBreakCondition(
    name="BreakIfHasMoreIsFalse",
    condition=BooleanEqualCondition(value=False),
    variable="{{%has_more%}}"
)

3. EmptyPropertyCondition

Description:

This condition stops pagination if a specific property in a data structure is considered "empty." It checks whether a property (such as a field in a JSON object or an attribute) contains an empty value. The value is optional in this condition.

YAML Example:

break_conditions:
  - name: BreakIfResponseIsEmpty
    condition:
      type: empty_property
    variable: "{{%response%}}"

Code Example:

break_condition = VariableBreakCondition(
    name="BreakIfResponseIsEmpty",
    condition=EmptyPropertyCondition(),
    variable="{{%response%}}"
)

Summary of Pagination-Specific Conditions

• FieldNotExistsCondition: Stops when a "next" token or field is not found in the API response.
• BooleanEqualCondition: Stops when a boolean flag (e.g., has_more) matches a specified value.
• EmptyPropertyCondition: Stops when a property or field is considered empty.

These break conditions provide flexibility for managing the pagination loop, ensuring the loop stops precisely when needed.

Example: FreshService API Pagination

Here’s an example using the FreshService API to paginate over a list of tickets until all tickets are retrieved:

# Prompt: Paginate over all the list of tickets in FreshService, until you reach the end of them
connector:
  name: FreshServiceConnector
  type: rest
  base_url: 'https://{FRESHSERVICE_DOMAIN}.freshservice.com/api'
  default_headers:
    Authorization: 'Basic {API_KEY}'
    Content-Type: "application/json"
  variables_storages:
    - name: results dir
      type: file_system
      path: storage/results/filesystem
  variables_metadata:
    tickets_output:
      storage_name: results dir
      format: json
    final_output_file:
      storage_name: results dir
      format: json

steps:
  - name: RetrieveTickets
    description: Retrieve a list of tickets with page-based pagination
    type: rest
    endpoint: "{{%BASE_URL%}}/v2/tickets"
    http_method: GET
    pagination:
      type: page
      parameters:
        - name: page
          value: 1
          increment_by: 1
        - name: per_page
          value: 30
      break_conditions:
        - name: BreakIfTicketsIs0
          condition:
            type: string_equal
            value: 0
          variable: "{{%tickets_output%}}"
      location: qs
      variables_output:
        - variable_name: tickets_output
          response_location: data
          variable_format: json
          overwrite_storage: true
          transformation_layers:
            - type: extract_json
              json_path: $.tickets
              from_type: json
        - variable_name: final_output_file
          response_location: data
          variable_format: json

Example: Jira API Offset-Based Pagination

Here’s an example using the Jira API to paginate through a list of issues using offset-based pagination:

steps:
  - name: RetrieveIssues
    description: Retrieve the List of Issues with integrated pagination
    type: rest
    endpoint: "{{%BASE_URL%}}/rest/api/2/search"
    http_method: GET
    pagination:
      type: offset
      parameters:
        - name: startAt  # Dynamic offset parameter name
          value: 0
          increment_by: 50
        - name: maxResults  # Dynamic limit parameter name
          value: 50
      break_conditions:
        - name: BreakIfSizeIs0
          condition:
            type: json_items_result_count
            value: 0
          variable: "{{%issues_output%}}"
      location: qs
    variables_output:
      - variable_name: issues_output
        response_location: data
        variable_format: json
        overwrite_storage: true
        transformation_layers:
          - type: extract_json
            json_path: $.issues.size
            from_type: json
      - variable_name: final_output_file
        response_location: data
        variable_format: json

Example: Spotify API Cursor-Based Pagination

Here’s an example using the Spotify API to paginate through a list of tracks using cursor-based pagination:

steps:
  - name: RetrieveTracks
    description: Retrieve the List of Tracks with integrated pagination
    type: rest
    endpoint: "{{%BASE_URL%}}/v1/me/tracks"
    http_method: GET
    pagination:
      type: cursor
      parameters:
        - name: "token"
          token_location: response
          token_format: json
          token_path: $.next
      break_conditions:
        - name: BreakIfNextIsNone
          condition:
            type: string_equal
            value: none
          variable: "{{%next_token%}}"
      location: qs
    variables_output:
      - variable_name: next_token
        response_location: data
        variable_format: json
        overwrite_storage: true
        transformation_layers:
          - type: extract_json
            json_path: $.next
            from_type: json
      - variable_name: final_output_file
        response_location: data
        variable_format: json

Explanation:

• Pagination: This example uses cursor-based pagination with the token being extracted from the response. The token_path is set to $.next, which retrieves the next page's token from the response.
• Break Conditions: The loop breaks when the next_token is none, stopping pagination.
• Variables: The result of the next_token is stored in next_token, and the final output file is saved as final_output_file.

6. Validation

When writing pagination YAML, it’s important to ensure the following:

• Consistency: The selected pagination type must be applied consistently across steps.
• Validation: Ensure that query parameters match pagination parameters and token paths align with token formats.
• Increment vs Cursor: For pagination of type `

pageoroffset, usePaginationIncrementParameter. Forcursor, usePaginationCursorParameter`.