Skip to main content

​
Installation

​
Installing Octopipe

Octopipe is designed to offer a seamless pipeline development experience, whether you are working locally or planning to deploy in the cloud. This guide walks you through the prerequisites, installation methods, and post-installation steps to ensure you have a fully functional setup.

​
Prerequisites

Before installing Octopipe, ensure that you have the following installed on your system:
  1. Python 3.8 or Higher Octopipe is built with Python, so make sure you have a compatible version installed.
  2. Node.js and npm Some parts of Octopipe’s tooling depend on Node.js. We recommend installing the latest LTS version.
  3. Docker (Recommended) Docker allows you to run Octopipe in a containerized environment for consistency and ease of deployment.
  4. Git To clone the repository and manage version control for any custom changes.
  5. A Modern Web Browser For accessing the Octopipe dashboard and documentation.

​
Installation Options

You can choose one of several methods to install Octopipe, depending on your development environment and preferences.

​
Option 1: Installation via Pip

Octopipe is available on PyPI. To install, run the following command: 
pip install octopipe
This command installs the Octopipe CLI and all required dependencies, enabling you to start managing pipelines immediately. Option 2: Installation from Source If you wish to work directly with the source code, follow these steps:
  1. Clone the Repository:
git clone https://github.com/your-org/octopipe.git
  1. Navigate to the Project Directory:
cd octopipe
  1. Install Python Dependencies:
pip install -r requirements.txt
  1. Install Node.js Dependencies (if applicable):
npm install
Option 3: Using Docker For those who prefer containerization, Docker offers a straightforward way to run Octopipe.
  1. Pull the Latest Docker Image:
docker pull your-org/octopipe:latest
  1. Run the Docker Container:
docker run -it -p 8000:8000 your-org/octopipe:latest
Post-Installation Steps After installation, there are a few steps to verify and configure your environment:
  1. Verify the Installation:
Run the following command to check the version: 
octopipe --version
The output should display the current Octopipe version.
  1. Configure Environment Variables:
Set up necessary environment variables. For example, to specify local development, you can set: 
export OCTOPIPE_ENV=local
  1. Display the Help Menu:
Familiarize yourself with the CLI by running: 
octopipe help
This command provides an overview of all available commands and options. Troubleshooting Tips • Dependency Issues: If you encounter errors regarding missing dependencies, verify that your Python and Node.js installations are up-to-date. • Permission Errors: Running the installation commands with proper privileges or within a virtual environment may resolve these issues. • Docker Troubles: Ensure Docker is running correctly on your system and that you have pulled the latest image. Additional Resources • Official Documentation: Visit our documentation website for more detailed guides and updates. • Community Support: Join our Slack or Discord channels for real-time assistance and to share experiences with other users. • GitHub Repository: For issue tracking or to contribute, visit our GitHub page. Summary By following this installation guide, you should now have a working setup of Octopipe. Whether you installed via pip, from source, or using Docker, you are now ready to explore and build your data pipelines. Enjoy your journey with Octopipe, and feel free to revisit this guide if you encounter any issues! 
---

### docs/getting-started/quickstart.mdx

```mdx
---
title: Quickstart Guide
---

# Quickstart with Octopipe

Welcome to the Octopipe quickstart guide. This document will help you set up and run your first pipeline in a few simple steps. Follow these instructions to quickly see Octopipe in action and start processing your data.

## Step 1: Initialize Your Project

Begin by initializing a new Octopipe project. This command sets up the basic structure and configuration files needed for your pipeline.

```bash
octopipe init --name my_pipeline --description "Pipeline for Sales Data"
• Tip: The —name flag defines your pipeline’s name, and —description provides context about its purpose. Step 2: Authenticate with Octopipe Before interacting with the platform, you need to authenticate. Use your API key to log in: 
octopipe login --api-key YOUR_API_KEY_HERE
• Note: Ensure your API key is active. Check the authentication section in the documentation if you encounter any issues. Step 3: Add a Data Source Configure your data source by adding a connector. For example, to add a sales API as a data source, run: 
octopipe source add --name sales_api --type api --option url=https://api.sales.com/data --option token=YOUR_TOKEN
• Explanation: • —name assigns a unique name to your data source. • —type specifies the kind of source (e.g., api, database, or file). • Additional options such as url and token provide connection details. Step 4: Add a Data Destination Now, set up where your data will be loaded. For instance, if you’re using PostgreSQL as your destination, run: 
octopipe destination add --name sales_db --type postgres --option host=localhost --option port=5432 --option user=dbuser --option password=secret --option database=sales
• Detail: This command configures the database connection by specifying host, port, credentials, and the target database name. Step 5: Define a Transformation Create a transformation that maps the type safe API schema to the labeled database schema. You can define this mapping using a schema file: 
octopipe transform add --name sales_transform --source sales_api --destination sales_db --schema-file ./schemas/sales_schema.json
• Clarification: The transformation aligns API fields with database fields, ensuring data consistency during the load process. Step 6: Create the Pipeline With your components in place, create the pipeline that orchestrates these elements: 
octopipe pipeline create --name daily_sales --source sales_api --destination sales_db --transform sales_transform --schedule "0 0 * * *"
• Schedule: The cron expression “0 0 * * *” schedules the pipeline to run daily at midnight. Step 7: Start the Pipeline Launch your pipeline to begin processing data: 
octopipe start daily_sales
• Observation: A successful start will display confirmation messages and log entries indicating that data processing has begun. Step 8: Monitor the Pipeline To view real-time updates and logs, use the following command: 
octopipe logs daily_sales --follow
• Tip: The —follow option streams live logs to your terminal, allowing you to monitor progress and troubleshoot if needed. Final Thoughts Congratulations! You have now set up and started your first Octopipe pipeline. This quickstart guide covered initialization, authentication, source and destination configuration, transformation setup, pipeline creation, and monitoring. If you need additional assistance, refer to the detailed documentation or connect with the community for support. Enjoy building robust data pipelines with Octopipe!