Pagination

Step-by-Step Guide: Configuring Pagination in Blueprints

This document provides a clear, step-by-step explanation on how to set up pagination in a Blueprint. Follow these instructions to ensure your Blueprint handles pagination effectively.

1. Understanding Pagination in Blueprints

Pagination is essential when working with APIs that return large datasets. It ensures data is retrieved in manageable chunks by specifying how requests should be incremented and when to stop retrieving data. Blueprints support three primary pagination types:

• Offset-based: Uses an offset parameter to specify the starting point for data retrieval.
• Page-based: Uses a page parameter to specify which page of data to retrieve.
• Cursor-based: Uses a token or cursor to point to the next set of results.

2. Steps to Configure Pagination

Step 1: Define Pagination Type

Choose the appropriate pagination type for your API. Supported types include:

• Offset: Use for APIs that support offset-based pagination.
• Page: Use for APIs that support page numbers.
• Cursor: Use for APIs that use cursors or tokens for pagination.

Step 2: Set Pagination Parameters

Define the parameters required for pagination. Depending on the type, these parameters may include:

• Increment Parameter (Offset/Page):
  • Name: The name of the parameter (e.g., offset, page).
  • Value: The default value of the parameter (e.g., 0 for offset or 1 for page).
  • Increment By: The value by which the parameter should increase (e.g., 10 for offset or 1 for page).

YAML Example:

pagination:
  type: offset
  parameters:
    - name: offset
      value: 0
      increment_by: 10
  break_conditions:
    - name: BreakIfNoMoreData
      condition:
        type: field_not_exist
        value: true
      variable: "{{%next_page_token%}}"
  location: qs

• Cursor Parameter (Cursor):
  • Name: The name of the cursor parameter (e.g., cursor, token).
  • Value: The default value of the cursor (e.g., the starting token).
  • Token Location: Specifies where the token is extracted from (e.g., response or header).
  • Token Format: Specifies the format of the token (e.g., JSON).
  • Token Path: The path to extract the token (e.g., $.next_cursor).

YAML Example:

pagination:
  type: cursor
  parameters:
    - name: cursor
      value: ""
      token_location: response
      token_format: json
      token_path: "$.next_cursor"
  break_conditions:
    - name: BreakIfHasMoreIsFalse
      condition:
        type: boolean_equal
        value: false
      variable: "{{%has_more%}}"
  location: qs

Step 3: Specify Break Conditions

Define conditions to stop the pagination loop when the dataset is fully retrieved. Examples of break conditions include:

• Field Not Exists: Stops when a token or field (e.g., next_token) is missing.
• Boolean Equal: Stops when a boolean flag (e.g., has_more) is false.
• Empty Property: Stops when a response property is empty.

YAML Example:

pagination:
  break_conditions:
    - name: BreakIfResponseIsEmpty
      condition:
        type: empty_property
      variable: "{{%response%}}"

Step 4: Choose Pagination Data Location

Specify where the pagination parameters should be placed in the API request. Options include:

• Query String (QS): Adds parameters to the URL query string.
• URL: Appends parameters directly to the endpoint URL.
• Body: Includes parameters in the request body (used for POST requests).

Step 5: Validate Parameters and Logic

Ensure your configuration aligns with the API's specifications:

• Verify that query parameters match the expected pagination parameters.
• Confirm that the token path aligns with the token format.
• Validate the combination of cursor parameter fields to ensure correctness.

3. Best Practices for Pagination

• Increment Operation: Ensure that offset or page size increments are appropriate for the dataset size. If no increment value is provided, use a default increment (e.g., 10).
• Dynamic Parameter Names: Derive parameter names dynamically based on the API documentation.
• Consistency: Use the same pagination method throughout the workflow.
• Data Extraction: Clearly specify where and how to extract pagination data, such as through JSONPath or headers.
• Break Conditions: Always define break conditions to prevent infinite loops.
• Validation: Validate the configuration during setup to ensure it adheres to API requirements.

4. Full YAML Example

Below is a full YAML example demonstrating how pagination fits into a complete configuration:

connector:
  name: FreshServiceConnector
  type: rest
  base_url: 'https://{sub_domain}domain.com/api'
  default_headers:
    Authorization: 'Basic {API_KEY}'
    Content-Type: "application/json"
  variables_storages:
    - name: results dir
      type: file_system
      path: storage/results/filesystem
  variables_metadata:
    tickets_output:
      storage_name: results dir
      format: json
    final_output_file:
      storage_name: results dir
      format: json

steps:
  - name: RetrieveTickets
    description: Retrieve a list of tickets with page-based pagination
    type: rest
    endpoint: "{{%BASE_URL%}}/v2/tickets"
    http_method: GET
    pagination:
      type: page
      parameters:
        - name: page
          value: 1
          increment_by: 1
        - name: per_page
          value: 30
      break_conditions:
        - name: BreakIfTicketsIs0
          condition:
            type: string_equal
            value: 0
          variable: "{{%tickets_output%}}"
      location: qs
      variables_output:
        - variable_name: tickets_output
          response_location: data
          variable_format: json
          overwrite_storage: true
          transformation_layers:
            - type: extract_json
              json_path: $.tickets
              from_type: json
        - variable_name: final_output_file
          response_location: data
          variable_format: json

5. Summary of Pagination-Specific Conditions

• Field Not Exists: Stops when no "next" token is found.
• Boolean Equal: Stops when a boolean flag matches the specified value.
• Empty Property: Stops when a specific property in a data structure is empty.

By following this guide, you can effectively configure pagination in Blueprints, ensuring efficient data retrieval and seamless integration with APIs. For further assistance, refer to your API documentation or reach out to our support team.