> ## Documentation Index
> Fetch the complete documentation index at: https://docs.octopipe.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Architecture

# Architecture

# Octopipe Architecture Overview

Octopipe is built upon an opinionated architecture designed to streamline the process of creating and managing data pipelines. By leveraging best-of-breed tools and custom microservices, Octopipe ensures a robust, scalable, and developer-friendly experience. In this document, we explore each architectural component in detail.

## Core Components

1. **Meltano**
   * **Role:** Handles extraction and load processes.
   * **Details:** Meltano integrates with a variety of data sources, allowing Octopipe to pull data reliably. Its plugin system supports both extraction and load, ensuring that data flows seamlessly from source to destination.
   * **Benefits:** Simplifies ETL processes and enables rapid connector setup.
2. **Airflow**
   * **Role:** Orchestrates workflows and schedules pipeline execution.
   * **Details:** With Airflow, tasks can be scheduled, monitored, and retried automatically. This ensures that each pipeline step runs in the correct order and any failures are managed with built-in recovery mechanisms.
   * **Benefits:** Provides a robust scheduling framework and clear visualization of pipeline dependencies.
3. **S3**
   * **Role:** Serves as the primary storage layer.
   * **Details:** S3 is used to store intermediate data, logs, and artifacts. Its scalability and reliability make it an ideal choice for handling large volumes of data in a cost-effective manner.
   * **Benefits:** High availability, durability, and seamless integration with other AWS services.
4. **Kafka**
   * **Role:** Facilitates real-time data streaming and messaging.
   * **Details:** Kafka is used for streaming data between components and ensuring that messages are reliably passed through the pipeline. This is critical for handling real-time events and maintaining data consistency.
   * **Benefits:** Enables scalable, low-latency data streaming and decouples system components.
5. **Spark**
   * **Role:** Executes data transformations and processing tasks.
   * **Details:** Spark provides the computational power needed to handle complex data transformations at scale. In Octopipe, Spark executes the approved transform layers, ensuring data is processed efficiently.
   * **Benefits:** High performance and flexibility for both batch and streaming data processing.

## Supporting Microservices

In addition to these core components, Octopipe employs several custom microservices that enhance the overall pipeline creation process:

* **Schema Pulling Service:**

  Retrieves and aggregates destination schemas from various sources.

* **Schema Labeling Service:**

  Provides an interactive interface for users to iteratively label and refine destination schemas, ensuring accurate data mapping.

* **Connector Setup Service:**

  Automates the configuration of Meltano connectors for each data source, reducing manual setup time.

* **Type Safe API Generator:**

  Leverages both API calls and LLMs to derive a type-safe API for every connector, ensuring that the returned data adheres to strict type definitions.

* **LLM Wrapper:**

  Improves the quality of code generation prompts, helping to generate cleaner and more accurate transformation logic.

## Architectural Diagram

Below is a simplified diagram of the Octopipe architecture:

```yaml theme={null}
      +--------------------+
      |     Octopipe     |
      |  (User Interface) |
      +---------+---------+
                |
                v
      +--------------------+
      |  CLI & API Layer   |
      +---------+---------+
                |
                v
 +--------------+--------------+
 |       Core Microservices    |
 |  - Schema Pulling Service   |
 |  - Schema Labeling Service  |
 |  - Connector Setup Service  |
 |  - Type Safe API Generator  |
 |  - LLM Wrapper              |
 +--------------+--------------+
                |
                v
 +--------------+--------------+
 |       Data Orchestration    |
 |  - Meltano (Extract/Load)   |
 |  - Airflow (Scheduling)     |
 +--------------+--------------+
                |
                v
      +--------------------+
      |    Data Storage    |
      |         S3         |
      +--------------------+
                |
                v
      +--------------------+
      | Real-Time Messaging|
      |       Kafka        |
      +--------------------+
                |
                v
      +--------------------+
      |   Data Processing  |
      |       Spark        |
      +--------------------+
```

## Key Architectural Principles

* **Modularity:**

  Each component and microservice is designed to perform a specific function, making the system easier to maintain and scale.

* **Scalability:**

  By leveraging distributed systems like Kafka and Spark, Octopipe can handle increases in data volume without significant re-architecture.

* **Developer Experience:**

  The integration of CLI tools and detailed logging ensures that developers can quickly set up, test, and monitor pipelines in both local and production environments.

* **Automation:**

  Many repetitive tasks, such as connector setup and API generation, are automated, reducing the potential for human error and speeding up development cycles.

## Conclusion

The Octopipe architecture is a carefully crafted blend of modern data tools and custom services that work together to provide a seamless pipeline creation experience. Whether you are developing locally or deploying to the cloud, Octopipe’s architecture is designed to scale with your data needs while ensuring a high level of reliability and performance.