Core Concepts

BSPump is built around a composable architecture where data flows through pipelines consisting of sources, processors, and sinks. Understanding these core concepts is essential for building effective data pipelines.

• Pipeline
  • Basic Pipeline Structure
  • Pipeline Lifecycle
  • Pipeline Methods
  • Throttling and Backpressure
  • Pipeline Configuration
  • Working with Multiple Pipelines
• Source
  • Source Types
  • Basic Source
  • TriggerSource
  • Built-in Sources
  • Source Configuration
  • Context
• Processor
  • Basic Processor
  • Filtering Events
  • Async Processors
  • Generator (1-to-Many)
  • Built-in Processors
  • Processor Chains
  • Error Handling
• Sink
  • Basic Sink
  • Async Sinks
  • Built-in Sinks
  • Batching
  • Backpressure
  • Error Handling
  • Multiple Outputs
  • Sink Configuration
• Connection
  • Why Use Connections?
  • Creating a Connection
  • Using Connections in Pipelines
  • Jupyter Connection Registration
  • Built-in Connections
  • Connection Configuration
  • Connection Lifecycle
  • Custom Connections
• Lookup
  • What are Lookups?
  • Using Lookups
  • Built-in Lookup Types
  • Loading from Files
  • Database-backed Lookups
  • Registering Lookups
  • Jupyter Lookup Registration
  • Lookup Updates
  • Lookup Configuration
  • Custom Lookups
• Trigger
  • What are Triggers?
  • Using Triggers
  • Built-in Triggers
  • Multiple Triggers
  • Trigger with Arguments
  • Cron Expression Reference
  • Custom Triggers

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                        Application                          │
│  ┌───────────────────────────────────────────────────────┐  │
│  │                      Pipeline                         │  │
│  │                                                       │  │
│  │   ┌────────┐   ┌───────────┐   ┌───────────┐   ┌────┐ │  │
│  │   │ Source │──▶│ Processor │──▶│ Processor │──▶│Sink│ │  │
│  │   └────────┘   └───────────┘   └───────────┘   └────┘ │  │
│  │                                                       │  │
│  └───────────────────────────────────────────────────────┘  │
│                                                             │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐          │
│  │ Connection  │  │   Lookup    │  │   Trigger   │          │
│  └─────────────┘  └─────────────┘  └─────────────┘          │
└─────────────────────────────────────────────────────────────┘

Key Components

Pipeline

The core abstraction that chains components together. Events flow from source through processors to the sink.

Source

Entry point for data. Sources can pull data (polling) or receive data (push-based like webhooks). See Source.

Processor

Transforms, filters, or enriches events. Multiple processors can be chained together. See Processor.

Sink

Exit point for data. Sinks write events to external systems, files, or other destinations. See Sink.

Connection

Shared, reusable connections to external systems (databases, message queues, etc.). See Connection.

Lookup

Data enrichment tables that can be used to add context to events. See Lookup.

Trigger

Controls when sources produce events (cron schedules, pub/sub, etc.). See Trigger.

Event Flow

Events flow through the pipeline in a linear fashion:

1. Source generates or receives an event

2. Event passes through each Processor in order

3. Each processor can transform, filter, or split the event

4. Sink receives the final event and outputs it

Events can be any Python object, but are commonly:

• Bytes (raw data)

• Dictionaries (structured data)

• Dataclasses or typed objects

Async-First Design

BSPump is built on Python’s asyncio, enabling:

• Non-blocking I/O operations

• High concurrency with minimal threads

• Efficient handling of many simultaneous connections

• Natural integration with async libraries