Common Patterns

This section covers common patterns for building data pipelines with BSPump. These patterns are based on real-world usage in production environments.

• Webhook to Kafka
  • Use Cases
  • Architecture
  • Jupyter Implementation
  • Standalone Application
  • Configuration
  • Best Practices
• Kafka Processing Pipeline
  • Use Cases
  • Architecture
  • Jupyter Implementation
  • Standalone Application
  • Configuration
  • Multiple Output Topics
  • Consumer Groups
  • Best Practices
• Custom Event Source
  • Use Cases
  • Basic Custom Source
  • Polling REST API
  • Using Thread Pools
  • TriggerSource for Scheduled Polling
  • Jupyter Implementation
  • Configuration
  • Best Practices
• Scheduled Tasks
  • Use Cases
  • Architecture
  • Jupyter Implementation
  • Standalone Application
  • Multiple Triggers
  • Batch Processing with Scheduled Source
  • Cron Expression Examples
  • Configuration
  • Best Practices
• Error Recovery
  • Use Cases
  • Kafka for Restart Safety
  • Implementation
  • Dead-Letter Queue Pattern
  • Retry with Exponential Backoff
  • Circuit Breaker Pattern
  • Graceful Shutdown
  • Configuration
  • Best Practices

Overview

The patterns in this section demonstrate:

• Webhook to Kafka: Receiving HTTP webhooks and forwarding to Kafka

• Kafka Processing: Consuming, processing, and producing to Kafka

• Custom Event Source: Building custom sources with thread pools

• Scheduled Tasks: Running periodic jobs with CronTrigger

• Error Recovery: Using Kafka for restart safety

Each pattern includes complete, runnable code examples.

Pattern Selection Guide

Use Case	Pattern
Receive external HTTP data	Webhook to Kafka
Transform streaming data	Kafka Processing Pipeline
Poll external systems	Custom Event Source
Run periodic jobs	Scheduled Tasks
Handle failures gracefully	Error Recovery