- 4 Minutes to read
- Print
- DarkLight
- PDF
Pagination
- 4 Minutes to read
- Print
- DarkLight
- PDF
Blueprint and Copilot Now Available in private preview!
We are excited to announce that our new Copilot and Blueprint features are now available to a select group of beta customers! This exclusive opportunity allows you to explore and test the latest capabilities of Blueprint before its official release.
If you're interested in joining the beta program, click here to request access.
Step-by-Step Guide: Configuring Pagination in Blueprints
This document provides a clear, step-by-step explanation on how to set up pagination in a Blueprint. Follow these instructions to ensure your Blueprint handles pagination effectively.
1. Understanding Pagination in Blueprints
Pagination is essential when working with APIs that return large datasets. It ensures data is retrieved in manageable chunks by specifying how requests should be incremented and when to stop retrieving data. Blueprints support three primary pagination types:
- Offset-based: Uses an offset parameter to specify the starting point for data retrieval.
- Page-based: Uses a page parameter to specify which page of data to retrieve.
- Cursor-based: Uses a token or cursor to point to the next set of results.
2. Steps to Configure Pagination
Step 1: Define Pagination Type
Choose the appropriate pagination type for your API. Supported types include:
- Offset: Use for APIs that support offset-based pagination.
- Page: Use for APIs that support page numbers.
- Cursor: Use for APIs that use cursors or tokens for pagination.
Step 2: Set Pagination Parameters
Define the parameters required for pagination. Depending on the type, these parameters may include:
- Increment Parameter (Offset/Page):
- Name: The name of the parameter (e.g.,
offset
,page
). - Value: The default value of the parameter (e.g.,
0
for offset or1
for page). - Increment By: The value by which the parameter should increase (e.g.,
10
for offset or1
for page).
- Name: The name of the parameter (e.g.,
YAML Example:
pagination:
type: offset
parameters:
- name: offset
value: 0
increment_by: 10
break_conditions:
- name: BreakIfNoMoreData
condition:
type: field_not_exist
value: true
variable: "{{%next_page_token%}}"
location: qs
- Cursor Parameter (Cursor):
- Name: The name of the cursor parameter (e.g.,
cursor
,token
). - Value: The default value of the cursor (e.g., the starting token).
- Token Location: Specifies where the token is extracted from (e.g.,
response
orheader
). - Token Format: Specifies the format of the token (e.g.,
JSON
). - Token Path: The path to extract the token (e.g.,
$.next_cursor
).
- Name: The name of the cursor parameter (e.g.,
YAML Example:
pagination:
type: cursor
parameters:
- name: cursor
value: ""
token_location: response
token_format: json
token_path: "$.next_cursor"
break_conditions:
- name: BreakIfHasMoreIsFalse
condition:
type: boolean_equal
value: false
variable: "{{%has_more%}}"
location: qs
Step 3: Specify Break Conditions
Define conditions to stop the pagination loop when the dataset is fully retrieved. Examples of break conditions include:
- Field Not Exists: Stops when a token or field (e.g.,
next_token
) is missing. - Boolean Equal: Stops when a boolean flag (e.g.,
has_more
) isfalse
. - Empty Property: Stops when a response property is empty.
YAML Example:
pagination:
break_conditions:
- name: BreakIfResponseIsEmpty
condition:
type: empty_property
variable: "{{%response%}}"
Step 4: Choose Pagination Data Location
Specify where the pagination parameters should be placed in the API request. Options include:
- Query String (QS): Adds parameters to the URL query string.
- URL: Appends parameters directly to the endpoint URL.
- Body: Includes parameters in the request body (used for POST requests).
Step 5: Validate Parameters and Logic
Ensure your configuration aligns with the API's specifications:
- Verify that query parameters match the expected pagination parameters.
- Confirm that the token path aligns with the token format.
- Validate the combination of cursor parameter fields to ensure correctness.
3. Best Practices for Pagination
- Increment Operation: Ensure that offset or page size increments are appropriate for the dataset size. If no increment value is provided, use a default increment (e.g.,
10
). - Dynamic Parameter Names: Derive parameter names dynamically based on the API documentation.
- Consistency: Use the same pagination method throughout the workflow.
- Data Extraction: Clearly specify where and how to extract pagination data, such as through JSONPath or headers.
- Break Conditions: Always define break conditions to prevent infinite loops.
- Validation: Validate the configuration during setup to ensure it adheres to API requirements.
4. Full YAML Example
Below is a full YAML example demonstrating how pagination fits into a complete configuration:
connector:
name: FreshServiceConnector
type: rest
base_url: 'https://{sub_domain}domain.com/api'
default_headers:
Authorization: 'Basic {API_KEY}'
Content-Type: "application/json"
variables_storages:
- name: results dir
type: file_system
path: storage/results/filesystem
variables_metadata:
tickets_output:
storage_name: results dir
format: json
final_output_file:
storage_name: results dir
format: json
steps:
- name: RetrieveTickets
description: Retrieve a list of tickets with page-based pagination
type: rest
endpoint: "{{%BASE_URL%}}/v2/tickets"
http_method: GET
pagination:
type: page
parameters:
- name: page
value: 1
increment_by: 1
- name: per_page
value: 30
break_conditions:
- name: BreakIfTicketsIs0
condition:
type: string_equal
value: 0
variable: "{{%tickets_output%}}"
location: qs
variables_output:
- variable_name: tickets_output
response_location: data
variable_format: json
overwrite_storage: true
transformation_layers:
- type: extract_json
json_path: $.tickets
from_type: json
- variable_name: final_output_file
response_location: data
variable_format: json
5. Summary of Pagination-Specific Conditions
- Field Not Exists: Stops when no "next" token is found.
- Boolean Equal: Stops when a boolean flag matches the specified value.
- Empty Property: Stops when a specific property in a data structure is empty.
By following this guide, you can effectively configure pagination in Blueprints, ensuring efficient data retrieval and seamless integration with APIs. For further assistance, refer to your API documentation or reach out to our support team.