​
Octopipe Glossary

This glossary provides definitions for key terms used throughout the Octopipe documentation. Understanding these terms is essential for effectively using and troubleshooting the system.

​
Terms and Definitions

  • API (Application Programming Interface): A set of protocols and tools for building software applications. In Octopipe, APIs are used to interact with external data sources and connectors.

  • Connector: A module that establishes a connection to a specific data source or destination. Connectors handle the details of data extraction and loading.

  • Type Safe API: An automatically generated API that enforces strict data types for all interactions. This ensures that data is handled consistently and reduces runtime errors.

  • Transform Layer: The component that maps the type safe API schema to the labeled database schema. It converts data into a format that is suitable for the destination system.

  • Meltano: A tool integrated within Octopipe for managing data extraction and load processes. It simplifies the integration of various data sources.

  • Airflow: A workflow orchestration tool used by Octopipe to schedule and monitor pipelines. Airflow manages task dependencies and execution order.

  • S3 (Simple Storage Service): A scalable storage solution provided by AWS, often used for storing intermediate data and logs.

  • Kafka: A distributed event streaming platform used for real-time data pipelines and messaging. In Octopipe, Kafka helps manage streaming data.

  • Spark: A unified analytics engine for large-scale data processing. Spark is used to execute transformation logic on large datasets.

  • LLM (Large Language Model): An AI model used to assist in generating type safe APIs and transformation code. The LLM derives data types from API responses to ensure consistency.

  • CLI (Command-Line Interface): The primary way developers interact with Octopipe. The CLI provides commands for project initialization, pipeline management, and configuration.

  • Cron Expression: A string representing a schedule, used to configure when pipelines should run. Cron expressions follow a standard syntax (e.g., "0 0 * * *" for daily execution).

  • Local Development: The practice of developing and testing pipelines on a local machine before deploying them to production. Octopipe supports an excellent local development experience.

  • Cloud Deployment: Running Octopipe on cloud infrastructure, which may offer additional scalability and managed services.

  • Configuration File: A file in JSON or YAML format that stores settings for Octopipe. Configuration files are used to maintain consistent settings across environments.

  • Environment Variables: Variables set in your operating system that Octopipe can use for configuration, especially for sensitive data like API keys.

  • Verbose Mode: An option that provides detailed logging output. This is useful for debugging and understanding the inner workings of commands.

​
Additional Concepts

  • Error Handling: Mechanisms built into Octopipe to manage and recover from errors during pipeline execution. Detailed logs help identify and fix issues.

  • Scheduling: The process of setting up pipelines to run at specific intervals using cron expressions. Scheduling is handled by Airflow.

  • Dependency Management: Ensuring that all parts of a pipeline (sources, transformations, destinations) are properly configured and connected before execution.

​
Conclusion

This glossary is designed to clarify the key terms and concepts used in Octopipe. Familiarity with these terms will help you navigate the documentation and use the platform more effectively.