​
Architecture

​
Octopipe Architecture Overview

Octopipe is built upon an opinionated architecture designed to streamline the process of creating and managing data pipelines. By leveraging best-of-breed tools and custom microservices, Octopipe ensures a robust, scalable, and developer-friendly experience. In this document, we explore each architectural component in detail.

​
Core Components

  1. Meltano
    • Role: Handles extraction and load processes.
    • Details: Meltano integrates with a variety of data sources, allowing Octopipe to pull data reliably. Its plugin system supports both extraction and load, ensuring that data flows seamlessly from source to destination.
    • Benefits: Simplifies ETL processes and enables rapid connector setup.
  2. Airflow
    • Role: Orchestrates workflows and schedules pipeline execution.
    • Details: With Airflow, tasks can be scheduled, monitored, and retried automatically. This ensures that each pipeline step runs in the correct order and any failures are managed with built-in recovery mechanisms.
    • Benefits: Provides a robust scheduling framework and clear visualization of pipeline dependencies.
  3. S3
    • Role: Serves as the primary storage layer.
    • Details: S3 is used to store intermediate data, logs, and artifacts. Its scalability and reliability make it an ideal choice for handling large volumes of data in a cost-effective manner.
    • Benefits: High availability, durability, and seamless integration with other AWS services.
  4. Kafka
    • Role: Facilitates real-time data streaming and messaging.
    • Details: Kafka is used for streaming data between components and ensuring that messages are reliably passed through the pipeline. This is critical for handling real-time events and maintaining data consistency.
    • Benefits: Enables scalable, low-latency data streaming and decouples system components.
  5. Spark
    • Role: Executes data transformations and processing tasks.
    • Details: Spark provides the computational power needed to handle complex data transformations at scale. In Octopipe, Spark executes the approved transform layers, ensuring data is processed efficiently.
    • Benefits: High performance and flexibility for both batch and streaming data processing.

​
Supporting Microservices

In addition to these core components, Octopipe employs several custom microservices that enhance the overall pipeline creation process:

  • Schema Pulling Service:

    Retrieves and aggregates destination schemas from various sources.

  • Schema Labeling Service:

    Provides an interactive interface for users to iteratively label and refine destination schemas, ensuring accurate data mapping.

  • Connector Setup Service:

    Automates the configuration of Meltano connectors for each data source, reducing manual setup time.

  • Type Safe API Generator:

    Leverages both API calls and LLMs to derive a type-safe API for every connector, ensuring that the returned data adheres to strict type definitions.

  • LLM Wrapper:

    Improves the quality of code generation prompts, helping to generate cleaner and more accurate transformation logic.

​
Architectural Diagram

Below is a simplified diagram of the Octopipe architecture:

      +--------------------+
      |     Octopipe     |
      |  (User Interface) |
      +---------+---------+
                |
                v
      +--------------------+
      |  CLI & API Layer   |
      +---------+---------+
                |
                v
 +--------------+--------------+
 |       Core Microservices    |
 |  - Schema Pulling Service   |
 |  - Schema Labeling Service  |
 |  - Connector Setup Service  |
 |  - Type Safe API Generator  |
 |  - LLM Wrapper              |
 +--------------+--------------+
                |
                v
 +--------------+--------------+
 |       Data Orchestration    |
 |  - Meltano (Extract/Load)   |
 |  - Airflow (Scheduling)     |
 +--------------+--------------+
                |
                v
      +--------------------+
      |    Data Storage    |
      |         S3         |
      +--------------------+
                |
                v
      +--------------------+
      | Real-Time Messaging|
      |       Kafka        |
      +--------------------+
                |
                v
      +--------------------+
      |   Data Processing  |
      |       Spark        |
      +--------------------+

​
Key Architectural Principles

  • Modularity:

    Each component and microservice is designed to perform a specific function, making the system easier to maintain and scale.

  • Scalability:

    By leveraging distributed systems like Kafka and Spark, Octopipe can handle increases in data volume without significant re-architecture.

  • Developer Experience:

    The integration of CLI tools and detailed logging ensures that developers can quickly set up, test, and monitor pipelines in both local and production environments.

  • Automation:

    Many repetitive tasks, such as connector setup and API generation, are automated, reducing the potential for human error and speeding up development cycles.

​
Conclusion

The Octopipe architecture is a carefully crafted blend of modern data tools and custom services that work together to provide a seamless pipeline creation experience. Whether you are developing locally or deploying to the cloud, Octopipe’s architecture is designed to scale with your data needs while ensuring a high level of reliability and performance.