​

octopipe source Command Reference

The octopipe source command is used to manage data sources within your Octopipe project. It allows you to add, list, update, and remove data sources that serve as the entry points for your pipelines.

​

Purpose

• Data Ingestion: Configure how and from where Octopipe retrieves data.
• Central Management: Maintain a consistent and standardized list of sources across your pipelines.

​

Usage

octopipe source <subcommand> [options]

Subcommands

add

• Purpose: Add a new data source.

• Usage Example:

octopipe source add --name sales_api --type api --option url=https://api.sales.com/data --option token=SALES_TOKEN

• Options:

• --name <source_name>: Unique name for the data source.

• --type <source_type>: Type of source (e.g., api, database, file).

• --config-file <path>: Path to a configuration file for complex setups.

• --option <key>=<value>: Additional options; can be repeated for multiple key-value pairs.

list

• Purpose: List all configured data sources.

• Usage Example:

octopipe source list

update

• Purpose: Update settings for an existing source.

• Usage Example:

octopipe source update sales_api --option token=NEW_TOKEN

remove

• Purpose: Remove an existing data source.

• Usage Example:

octopipe source remove sales_api

Detailed Behavior

• Validation:

When adding or updating a source, Octopipe validates the provided parameters to ensure connectivity.

• Configuration Files:

For complex sources, a configuration file can be provided to streamline the setup.

• Dynamic Updates:

Changes to a source can take effect immediately without restarting the entire pipeline system.

Examples

Adding an API Data Source

octopipe source add --name sales_api --type api --option url=https://api.sales.com/data --option token=SALES_TOKEN

Adding a Database Source

octopipe source add --name customer_db --type database --option host=localhost --option port=5432 --option user=dbuser --option password=secret --option database=customers

Listing Data Sources

octopipe source list

Updating a Source

octopipe source update sales_api --option token=NEW_TOKEN

Removing a Source

octopipe source remove customer_db

Best Practices

• Consistent Naming:

Use clear and descriptive names for your sources to avoid confusion.

• Secure Options:

Ensure that sensitive information such as tokens or passwords are handled securely, preferably using environment variables.

• Configuration Files:

For sources with many options, prefer a configuration file to keep command lines clean and maintainable.

Troubleshooting

• Connection Issues:

If a source fails to connect, verify the provided URL, token, or credentials.

• Configuration Errors:

Check for typos in the options or in the configuration file format.

• Listing Problems:

If the list command does not return expected results, ensure that the sources have been added correctly and check for any silent errors in the logs.

Conclusion

The octopipe source command is a vital tool for managing the various data entry points into your pipelines. By following the examples and best practices outlined in this guide, you can efficiently set up and maintain your data sources.