Unlock all the data
that powers your
product featuresML trainingdata platform
Say goodbye to writing, maintaining, and scaling your own API integrations
with our declarative code-first data integration engine
Loved by developers, data scientists & ML engineers
You build the features
we get you the data
Unlock 600+ APIs and DBs with a single YAML file and CLI.
Realize your wildest data and ML-powered product ideas.
Sources
Destinations
SaaS API
Database
File
Custom Source
Database
Vector DB
Data Warehouse
Data LakeConfidently
code-first
Declaratively define your data pipelines inside your codebase and use common DevOps workflows. It's like Terraform for data integration.
Set up for
scale
Manage 100s of sources, 1000s of customer pipelines, and PBs of data with ease using Meltano's declarative YAML config, CLI, and API.
Endlessly
extensible
Build and improve connectors yourself, process data using your own Python scripts and data tools like dbt, or even contribute to Meltano itself.
Made for
ML & AI
Store your data somewhere your ML code can find it, or generate embeddings from inside your pipeline and store them in a vector database.
All of your customers’ data right where you need it
100% of your data sources & destinations just a keystroke away: Databases, SaaS APIs, raw files, and anything niche or internal
Programmable
Integrate deeply with your product
- Let your users directly connect data sources and manage pipelines from your application code
- Define pipeline templates and dynamically instantiate them based on API credentials in your database
- Configure connectors at runtime using environment variables or a dynamically generated config file
meltano.yml
plugins:extractors:- name: tap-shopify- name: tap-postgres--shopify-configsinherit_from: tap-postgresselect:- shopify_configs.*loaders:- name: target-snowflakeschedules:- name: sync-all-shopifyinterval: @hourlyconfig_source:tap-shopify: tap-postgres--shopify-configsextractor: tap-shopifyloader: target-snowflake
Testable
Make changes with confidence
- Safely experiment with new pipelines, connectors, and configuration changes on your local machine
- Manually verify the impact of any changes in isolated feature-branch and staging environments
- Automatically test your pipelines in CI, and only deploy to production once everything is green
Terminal
git clone https://github.com/my-org/my-meltano-project.gitgit checkout -b update-postgresmeltano config tap-postgres set host new-host.examplemeltano select tap-postgres new-table "*"meltano run tap-postgres target-snowflakegit commit -am "Update postgres host and table"git push origingh pr create
Versionable
Increase productivity and collaboration
- Store your pipelines and their configuration in Git and know exactly what is live at any time
- Roll back to the last known-good configuration if a change accidentally broke something
- Propose changes with confidence and get feedback from your team in pull requests
meltano.yml
plugins:extractors:- name: tap-postgresconfig:host: pg.example.comuser: adminpassword: $PG_PASSWORDselect:- users.*loaders:- name: target-snowflakeschedules:- name: app-to-dwinterval: @hourlyextractor: tap-postgresloader: target-snowflake
Customizable
Manage all your pipelines in one place
- Custom connectors for niche or internal sources, built with our Meltano SDK
- Custom forks of existing connectors with your own modifications
- Your Python scripts and data tools like dbt
Terminal
meltano add extractor tap-postgresmeltano add loader target-snowflakecookiecutter https://github.com/meltano/sdk \--directory="cookiecutter/tap-template# source_name: my-apimeltano add --custom extractor tap-my-apimeltano add utility dbt-snowflakemeltano add --custom utility my-script.pymeltano run tap-postgres target-snowflake dbt-snowflakemeltano run tap-my-api target-snowflake my-script.py
Transparent
Know exactly what's going on
- Get the full view with detailed pipeline logs
- Inspect the source code of any connector, modify them to your liking, or debug and fix an issue faster than any support agent
- Centrally manage your configuration so anyone can suggest changes
Terminal
export MELTANO_LOG_LEVEL=debugmeltano run tap-github target-duckdbcat .meltano/logs/elt/github-to-duckdb-hourly/*/*.loggit clone https://github.com/MeltanoLabs/tap-github.gitgit clone https://github.com/jwills/target-duckdb.gitmeltano add --custom extractor tap-github-fork# pip_url: https://github.com/my-org/tap-github.gitmeltano run tap-github-fork target-duckdb
Your one-stop shop for
customer data integrations
Don't limit yourself to data under your direct control.
Build powerful features on all customer data, wherever it may be.
SaaS apps and APIs
Choose from the largest connector library of any EL tool, and modify or fix connector behavior without waiting for support. Ensure data security using in-flight filtering and hashing of PII, and debug using detailed pipeline logs and alerting.
Databases and warehouses
Painlessly move data in and out of any relational, vector, or columnar database: fast imports of large batches plus key-based and log-based (CDC) incremental replication with automatic schema migration. Ensure data security using in-flight filtering and hashing of PII.
Files and spreadsheets
Extract data from any common or custom file format, stored in S3, (S)FTP, URLs, or cloud drives – with automatic schema migration. Ensure data security using in-flight filtering and hashing of PII, and debug using detailed pipeline logs and alerting.
Internal and niche APIs
Quickly build a connector for any API using the Meltano SDK: built-in error and rate limit handling plus zero-effort incremental replication. Develop, test, and iterate locally and in staging, and debug using detailed pipeline logs and alerting.
