What Is Meltano?

Meltano is an open source platform for building, running & orchestrating ELT pipelines built out of Singer taps and targets and dbt models, that you can run locally or easily deploy in production.

Mission

Our goal is to make the power of data integration available to all by building a true open source alternative to existing proprietary hosted EL(T) solutions, in terms of ease of use, reliability, and quantity and quality of supported data sources.

For more context, read the following blog post: Why we are building an open source platform for ELT pipelines.

Focus

We have decided to focus on the following personas and use cases, in order of priority, to help us determine what improvements to prioritize building, and what community contributions to accept:

  1. Data engineers already using Singer taps and targets

    • They are running Singer-based data pipelines in production already, likely using a hand-rolled hard-to-reproduce deployment/orchestration strategy.
    • They have experience building and maintaining taps and targets.
    • They are comfortable using a CLI and want to manage their Meltano project like a software engineering project, with all of the benefits of version control and CI/CD.
    • They want to have the option of using a web-based UI to monitor (and possibly manage) their pipelines.
    • Meltano can learn from their experience and provide tooling and documentation to make it easier for them and new users to manage and deploy Singer-based data pipelines and build and maintain new and existing Singer taps and targets.
  2. Data engineers or one-person data teams new to open source data pipelines

    • They may have come across Singer already, but haven't run Singer taps and targets in production yet.
    • Initially, they are primarily interested in using existing taps and targets instead of writing and maintaining their own, but if they run into issues, they would be comfortable attempting to fix bugs themselves and contributing the fixes upstream.
    • Later, they would not want to be limited by existing taps and targets and will want to learn how to build and maintain their own.
    • They are comfortable using a CLI and want to manage their Meltano project like a software engineering project, with all of the benefits of version control and CI/CD.
    • They want to have the option of using a web-based UI to monitor (and possibly manage) their pipelines.
    • They may also be interested in using Meltano to run dbt model-based transformations as part of their data pipelines.
  3. Hobbyists

    • They are capable of programming and comfortable using a CLI, but may not be data engineers.
    • They want to pull data from general or personal sources into a local database for personal use, e.g. basic analytics.
    • They want to run Meltano locally or in a non-production self-hosted environment.
    • They are primarily interested in using one or more specific existing extractors and loaders, but if they run into issues, they would be comfortable attempting to fix bugs themselves and contributing the fixes upstream.
    • They may also be interested in using Meltano's built-in analytics functionality, and may look into building the transformation and model plugins that support it.

History

Meltano, originally called BizOps, was founded inside GitLab in 2018 to serve the GitLab Data Team. It is maintained primarily by the Meltano team at GitLab, which continues to sponsor its development.

To learn more about Meltano's history from 2018 through 2020, read the following blog post: Revisiting the Meltano strategy: a return to our roots.