Blueprint Components and Configuration

User Guide: Creating Blueprints Manually

Welcome to the manual guide for creating Blueprints in Rivery. This document combines a step-by-step flow for users to create a successful connector and details the order of elements in the YAML configuration.

Step-by-Step Flow for Creating a Successful Connector

1. Configure the Connector

Start by defining the basic connector settings, including the name, type, and base URL of the API you want to use.

Connector:

connector:
  name: GitHubConnector
  type: rest
  base_url: 'https://api.github.com'

• name: Identifies the connector (e.g., GitHubConnector).
• type: Specifies the connector type as rest for RESTful API calls.
  • More types are coming soon
• base_url: The root endpoint for all API requests.

2. Define Default Headers

Add the necessary headers for authentication and API versioning according to the API documentation.

Headers:

default_headers:
  Authorization: 'Basic {YOUR_AUTH}'
  X-GitHub-Api-Version: '{YOUR_X_GITHUB_API_VERSION}'
  User-Agent: '{YOUR_USER_AGENT}'

• Authorization: Token-based authentication.
• X-GitHub-Api-Version: Specifies the API version.
• User-Agent: Identifies the client making the request.

3. Configure Variable Metadata

Blueprint uses variables for elements such as pagination or API responses.
Variable Metadata is where you should declare all variables that you plan to use in your YAML configuration. These variables are referenced in different parts of the workflow and must be set up here for proper operation.

YAML Structure:

variables_metadata:
  page_number:
    storage_name: results dir
    format: json
  last_result:
    storage_name: results dir
    format: json

• page_number: Tracks the current page for pagination.
• last_result: Holds the last API response data.

Declaring variables in this section ensures they are accessible and properly managed during the workflow execution.

4. Set Up Variable Storage

Define where variables and data will be stored. Rivery supports multiple storage types, and you can also integrate your own storage solution. For example:

YAML Structure:

variables_storages:
  - name: results dir
    type: file_system
    path: storage/results/filesystem

• results dir: Stores data in the filesystem at the specified path.
• results memory: Provides temporary in-memory storage for variables. Coming Soon

Custom Storage: Coming Soon

• Users can configure their own storage types based on the workflow’s requirements. For example, integrate with cloud storage services, databases, or other custom storage solutions to handle variables and results dynamically.

5. Build the Steps

Define the logical steps for retrieving and processing data.

##### Step A: Implement a Loop for Pagination

Create a while loop to paginate through API responses.

Step Structure:

- name: WhileLoopOverCommits
  description: "Get all commits from a GitHub repository"
  type: loop
  loop:
    type: while
    variable_name: page_number
    value: 1
    while_settings:
      operation_to_perform: add
      value_to_perform: 1
      max_iterations: 10000
    break_conditions:
      - name: BreakIfOutOfCommits
        condition:
          type: string_equal
          value: "[]"
        variable: "{{%last_result%}}"

• Break Condition: Stops if last_result is empty ("[]").
• Safety: Prevents infinite loops using max_iterations.

##### Step B: Retrieve API Data

Fetch data using an endpoint and store the response in variables.

YAML Structure:

steps:
  - name: Pagination
    description: Retrieve GitHub repository commits
    endpoint: "{{%BASE_URL%}}/repos/{YOUR_REPOSITORY_OWNER}/{YOUR_REPOSITORY}/commits"
    http_method: GET
    query_params:
      page: "{{%page_number%}}"
    variables_output:
      - variable_name: last_result
        response_location: data
        variable_format: json
        overwrite_storage: true
      - variable_name: final_output_file
        response_location: data
        transformation_layers:
          - type: extract_json
            json_path: $.[*].commit
            from_type: json

• Query Parameters: Adds the page number for pagination.
• Variables Output: Extracts and transforms data from the response.

##### Step C: Process Data in Loops

Use nested loops to process repositories, branches, and commits.

YAML Structure:

steps:
- name: LoopOverRepositories
  description: "Process each repository"
  loop:
    variable_name: "repositories"
    item_name: "repository"
    type: "data"
  steps:
    - name: GetBranchesForRepository
      endpoint: "{{%BASE_URL%}}/repos/{{%repository%}}/branches"
      method: GET
      type: rest
      variables_output:
        - variable_name: branches
          response_location: data
          variable_format: json
    - name: LoopOverBranches
      loop:
        variable_name: "branches"
        item_name: "branch"
        type: "data"
      steps:
        - name: GetCommitsForBranch
          endpoint: "{{%BASE_URL%}}/repos/{{%repository%}}/commits?sha={{%branch%}}"
          method: GET
          type: rest
          variables_output:
            - variable_name: sha_commits
              response_location: data
              variable_format: json

6. Add a Retry Strategy

Handle API errors gracefully by defining a retry mechanism.

YAML Structure:

steps:
  - name: FetchPosts
    description: "Fetch all posts from the API"
    endpoint: "{{%BASE_URL%}}/posts"
    http_method: GET
    expected_status_codes:
      - 200
    retry_strategy:
      400:
        max_attempts: 2
      429:
        max_attempts: 5

• Retries up to 2 times for 400 errors and 5 times for 429 errors.

Summary

1. Configure the Connector: Define its name, type, base URL, and default headers.
2. Configure Variable Metadata: Set up metadata for tracking and processing key variables, ensuring all required variables are declared for workflow use.
3. Set Up Variable Storage: Define storage solutions and integrate custom storage if needed.
4. Define Steps: Build the workflow logic, including loops, data retrieval, and transformations.
5. Add Retry Strategies: Ensure workflows handle errors and retries effectively.