​
Using the Octopipe CLI Tutorial

This tutorial is designed to help you become proficient with the Octopipe Command-Line Interface (CLI). You’ll learn about the various commands available and how to use them effectively to manage your pipelines.

​
Introduction

Octopipe’s CLI offers a rich set of commands to initialize projects, manage data sources and destinations, define transformations, and orchestrate pipelines. In this guide, we will cover:

  • Basic CLI navigation
  • Essential commands and their usage
  • Tips and best practices for effective CLI usage

​
Getting Started with the CLI

​
Verifying Installation

After installing Octopipe, check that the CLI is working by running:

octopipe --version

• Expected Output:

The current version of Octopipe will be displayed.

Accessing Help

For general help, use:

octopipe help

For command-specific help, append the command name. For example:

octopipe help init

Core Commands Overview

Initialization

Initialize a new project with:

octopipe init --name my_pipeline --description "Pipeline for sales data"

Authentication

Log in to access Octopipe’s features:

octopipe login --api-key YOUR_API_KEY

Log out when finished:

octopipe logout

Managing Data Sources and Destinations

Add a data source:

octopipe source add --name sales_api --type api --option url=https://api.sales.com/data --option token=YOUR_TOKEN

Add a data destination:

octopipe destination add --name sales_db --type postgres --option host=localhost --option port=5432 --option user=dbuser --option password=secret --option database=sales

Transformation and Pipeline Commands

Define a transformation:

octopipe transform add --name sales_transform --source sales_api --destination sales_db --schema-file ./schemas/sales_schema.json

Create a pipeline:

octopipe pipeline create --name daily_sales --source sales_api --destination sales_db --transform sales_transform --schedule "0 0 * * *"

Start the pipeline:

octopipe start daily_sales

Monitor logs:

octopipe logs daily_sales --follow

Advanced CLI Usage

• Verbose Mode:

Add —verbose to commands for detailed output. For example:

octopipe init --name verbose_pipeline --verbose

• Custom Configuration Files:

Use the —config-file flag to load custom settings:

octopipe init --name custom_pipeline --config-file ./config/custom.json

• Chaining Commands:

Combine commands in scripts to automate tasks like initialization and deployment.

Best Practices

• Keep Commands Modular:

Test each command individually to verify connectivity and correct configuration.

• Use Descriptive Names:

This helps maintain clarity when managing multiple pipelines.

• Regularly Update the CLI:

Stay current with the latest features and bug fixes.

Troubleshooting CLI Issues

• Command Not Found:

Verify your PATH environment variable includes the Octopipe binary.

• Authentication Errors:

Double-check your API key or username/password if login fails.

• Verbose Logging:

Use the —verbose flag to get more detailed error messages.

Conclusion

Mastering the Octopipe CLI is key to efficiently managing your data pipelines. This tutorial provided an overview of essential commands and best practices. Experiment with these commands and refer to our detailed reference documents for more advanced usage.

Happy CLI-ing!