Structure

Contents:

Introduction of GPT_graph’s structure.

  1. Closure: The foundational class that establishes hierarchical relationships.

  2. Component: The basic building blocks of the pipeline system.

  3. Pipeline: A specialized type of Component that can contain and manage other Components.

  4. Step: Executable units within a Pipeline. Generated by Components.

  5. Graph: Representations of the structure of connected steps/nodes.

  6. Parameter Handling: A flexible system for managing and prioritizing parameters.

Summary of Key Concepts:

  1. Closure:

    • Base class for Component and Session

    • Manages hierarchical relationships (i.e. Closure.contains is a list of other Closures)

  2. Component:

    • Three category: “class”, “static”, “method”

    • Defines input_schema, cache_schema, output_schema, and output_format

    • Uses Component.bindings/linkings to determine execution order of Components within a Pipeline.

  3. Pipeline:

    • Inherits from Component

    • Manages the connection and execution of Components

    • Uses the ‘|’ operator for linking Components

  4. Step:

    • Executable units within a Pipeline

    • Stored in Pipeline.sub_steps_q(priority queue) during runtime

  5. Graph:

    • Wrapper for networkx graph

    • Represents the structure of connected steps/ generated nodes

    • There are many two types of Graphs: node_graph and step_graph

  6. Parameter Handling:

    • Parameters are stored as dict in Component.params first. During run-time, they will be copied into Steps.

    • Implements a priority system for parameter values

    • Supports different parameter types and statuses

    • Allows for placeholder parameters

Important conventions:

  1. Each Component/ Pipeline have a method run that can be run independently. This facilitates debugging.

Shorthand Notations

  • Pipeline: pp

  • Component: cp