# Introduction

Meltano is an open source platform for building, running & orchestrating ELT pipelines made up of Singer taps and targets and dbt models, that you can run locally or easily deploy in production.

# Mission

Our goal is to make the power of data integration available to all by building a true open source alternative to existing proprietary hosted EL(T) solutions, in terms of ease of use, reliability, and quantity and quality of supported data sources.

For more context, read the following blog post: Why we are building an open source platform for ELT pipelines.

# Focus

We have decided to focus on the following personas and use cases, in order of priority, to help us determine what improvements to prioritize building, and what community contributions to accept:

  1. Data engineers already using Singer taps and targets

    • They are running Singer-based EL(T) pipelines in production already, likely using a hand-rolled hard-to-reproduce deployment/orchestration strategy.
    • They have experience building and maintaining taps and targets.
    • They are comfortable using a CLI and want to manage their Meltano project like a software engineering project, with all of the benefits of version control and CI/CD.
    • They want to have the option of using a web-based UI to monitor (and possibly manage) their pipelines.
    • Meltano can learn from their experience and provide tooling and documentation to make it easier for them and new users to manage and deploy Singer-based EL(T) pipelines and build and maintain new and existing Singer taps and targets.
  2. Data engineers or one-person data teams new to open source ELT

    • They may have come across Singer already, but haven't run Singer taps and targets in production yet.
    • Initially, they are primarily interested in using existing taps and targets instead of writing and maintaining their own, but if they run into issues, they would be comfortable attempting to fix bugs themselves and contributing the fixes upstream.
    • Later, they would not want to be limited by existing taps and targets and will want to learn how to build and maintain their own.
    • They are comfortable using a CLI and want to manage their Meltano project like a software engineering project, with all of the benefits of version control and CI/CD.
    • They want to have the option of using a web-based UI to monitor (and possibly manage) their pipelines.
    • They may also be interested in using Meltano to run dbt model-based transformations as part of their pipelines.
  3. Hobbyists

    • They are capable of programming and comfortable using a CLI, but may not be data engineers.
    • They want to pull data from general or personal sources into a local database for personal use, e.g. basic analytics.
    • They want to run Meltano locally or in a non-production self-hosted environment.
    • They are primarily interested in using one or more specific existing extractors and loaders, but if they run into issues, they would be comfortable attempting to fix bugs themselves and contributing the fixes upstream.
    • They may also be interested in using Meltano's built-in analytics functionality, and may look into building the transformation and model plugins that support it.

# Roadmap

Meltano is developed completely in the open on GitLab: https://gitlab.com/meltano/meltano. Our issue tracker, epics, and weekly milestones can be found there as well.

To get an idea of what the team and community are currently working on, check out the upcoming milestone's Development Flow board.

If you'd like to look further into the future, the Milestones board has a column for each upcoming weekly milestone.

Be aware that issue milestones serve more as a rough indication of relative priority than as hard deadlines, since short-term priorities can change quickly in response to community feedback, and it's hard to predict how much progress can be made in a week. Also note that issues labeled flow::To Do have higher priority and are more likely to be completed in a given week than those labeled flow::Triage, which are often moved to the next milestone at the end of the week, with issues already scheduled for the next week pushed out to make room for them.

# Contributing

Meltano is built for and by its community, and we welcome your contributions to our GitLab repository, which houses Meltano's core, CLI, UI, UI API, these docs, and the index of discoverable plugins, which feeds the lists of Extractors and Loaders that are supported out of the box.

To learn more about contributing to Meltano, refer to the Contributor Guide.

# History

Meltano, originally called BizOps, was founded inside GitLab in 2018 to serve the GitLab Data Team. It is maintained primarily by the Meltano team at GitLab, which continues to sponsor its development.

To learn more about Meltano's history from 2018 through 2020, read the following blog post: Revisiting the Meltano strategy: a return to our roots.