# Introduction

Meltano is ELT for the DataOps era: open source, self-hosted, CLI-first, debuggable, and extensible.

This page covers the project's Mission, Focus, Roadmap, History, and Team.

To find guides and references on other topics, use the Table of Contents in the sidebar.

# Mission

Our mission is to enable every organization to make the best decisions possible by becoming data-informed.

To achieve this mission we are building an open source platform for the complete DataOps lifecycle that is optimized for the happiness and productivity of Data Teams and Data Professionals. It integrates best-in-class open source components and enables teams to collaborate on data projects and pipelines more efficiently and with higher confidence.

Our focus has been on bringing these qualities to the first step in any data journey: integration and transformation, aka EL(T): Extract, Load, Transform, where traditional solutions are either off-the-shelf and near-impossible to extend, tweak, and debug, or fully custom and a pain to maintain.

We believe that data integration is ripe for commoditization, and are building towards a future in which fast and reliable open source connectors for every data source and destination in the world will be freely available, collectively maintained by a community of data engineers in consulting and at in-house data teams, and by API vendors themselves, just like we see with API client libraries today.

# Embracing Singer

To make this a reality, we have embraced Singer and are committed to providing its community and ecosystem with all of the tools and support needed for it to realize its full potential as the de-facto standard for open source connectors, to be used with Meltano or any other tool that supports them.

To further support the community and ecosystem, we have launched the MeltanoHub for Singer. This is the Singer equivalent of PyPI or Docker Hub, to give users (and tools) a central place to learn about the behavior, supported features, and maintenance status of all taps and targets in the ecosystem, which are currently scattered across Git repos and PyPI packages. As we continue to develop and enhance the MeltanoHub we will encourage decentralized maintenance of connectors to prevent individual organizations from becoming bottlenecks as the ecosystem grows, and will support the adoption of abandoned connectors by new maintainers.

With Meltano, we are providing a clear path to production with existing Singer taps and targets where there wasn't one before, thereby lowering the barrier to adoption and motivating people who run into an issue with an existing connector to debug it, contribute a fix, and see it through, instead of giving up. Meltano's index of discoverable extractors and loaders will grow as the number of Singer Taps and Targets grows.

In addition to the MeltanoHub, the SDK for Singer Taps and Targets enables developers to build connectors with all the bells and whistles without having to be become an expert on the spec. This further lowers the barrier to creating new connectors and contributing to existing ones, and will lead to increased reliability and consistency.

We've also created an interpretation of the Singer specification that aims to be clearer for newcomers to the project. We believe the spec is great in its current version, but confusing documentation has made it challenging for some to onboard to the community.

Last but not least, we intend to unlock the evolution of the Singer spec through a framework for the proposal and consideration of optional extensions to the spec that compatible tools, including the SDK and Meltano, can choose to support, while maintaining compatibility between all taps and targets.

# Focus

As described above, our vision for ELT in the DataOps era involves a decentralized ecosystem of open source connectors and a community of data engineers maintaining and contributing to these projects.

To make this happen, we are primarily focused on users and teams that have the potential and the necessary technical skills to become active participants in this community, rather than those who prefer a hands-off approach with off-the-shelf connectors maintained by a single (paid) provider.

In order of priority, our target personas are:

  1. Data consultants who may become implementation partners
    • They frequently write one-off (Python) scripts to extract data from niche sources that aren't supported by commercial EL(T) vendors.
    • They want to stop reinventing the wheel and standardize their approach to building reliable and reusable custom connectors.
    • They are willing to open source their custom connectors to share the maintenance burden with the community and, in return, get access to connectors built and maintained by other consulting firms and data engineers.
    • They want a consistent approach to extracting data from common sources (that are supported by commercial vendors) and niche ones (that aren't).
    • They want the ability to customize and fix bugs in connectors without having to wait on a vendor.
    • They want to save their clients money by self-hosting pipelines instead of being beholden to vendor pricing.
    • They want to give their clients the option of never having their sensitive data pass through systems out of their control for privacy or compliance reasons.
    • They want to manage their data projects like any other software engineering project, with all of the benefits of version control and CI/CD.
    • They want to offer clients a (whitelabel) web UI to set up their connections and manage their pipelines.
  2. Data engineers with a software development background
    • They are comfortable using a CLI and want to manage their data project like any other software engineering project, with all of the benefits of version control and CI/CD.
    • They recognize that connectors are like API client libraries, and are comfortable using open source options even if they may require some tweaking and contributing fixes upstream.
    • They want a consistent approach to extracting data from common sources (that are supported by commercial vendors) and niche ones (that aren't).
    • They want the ability to customize and fix bugs in connectors without having to wait on a vendor.
    • They may not want their sensitive data to pass through systems out of their control for privacy or compliance reasons.
    • They may want to extract data from region-specific sources and SaaS tools that are not supported by the US-centric commercial vendors.
    • They may have more development time to spend than money and would rather build and contribute to connectors than pay an expensive vendor.
  3. Developers of data products
    • They want to let their users connect sources directly to their product.
    • They want to be able to configure and run pipelines programmatically.
    • They want to leverage existing connectors so they can focus on what their product does with the data instead of getting access to it.
    • They want to open source any custom connectors they write to share the maintenance burden with the community.

# Roadmap

Meltano is developed completely in the open on GitLab: https://gitlab.com/meltano/meltano. Our issue tracker, epics, and weekly milestones can be found there as well.

To get an idea of what the team and community are currently working on, check out the upcoming milestone's Development Flow board.

If you'd like to look further into the future, the Milestones board has a column for each upcoming weekly milestone.

Be aware that issue milestones serve more as a rough indication of relative priority than as hard deadlines, since short-term priorities can change quickly in response to community feedback, and it's hard to predict how much progress can be made in a week. Also note that issues labeled flow::To Do have higher priority and are more likely to be completed in a given week than those labeled flow::Triage, which are often moved to the next milestone at the end of the week, with issues already scheduled for the next week pushed out to make room for them.

Below you will found our current roadmap. Each item should link to an Epic with the relevant issue linked to that epic. As this projects out into the future it is subject to change based on feedback. Don't see something you want on the roadmap? Make an issue and let us know!

Our focus for the rest of the year is going to mainly be in two categories:

  • Improving the overall quality of the Singer ecosystem
  • Making Meltano the best data integration (ELT) tool on the market

These efforts are the foundation of our larger vision to enable a full DataOps platform.

# September 2021

  • Add validator plugin type with Great Expectations and dbt test support - Epic
  • Composable Meltano Pipelines - Epic & Issue
  • MeltanoHub as SSOT for Meltano - Epic
  • Easy deployment of Meltano for local development - Epic & Issue
  • Decentralized Management of MeltanoHub Connectors - Epic & Issue

# October 2021

  • Composable Meltano Pipelines - Epic
  • Easy deployment of Meltano for production development - Epic
  • Telemetry Enhancements for Meltano, Meltano SDK, and MeltanoHub - Epic
  • Native Environments - Issue

# November 2021

  • Support non-Python plugins - Issue
  • Add Fast Sync / BATCH Messages to Meltano SDK - Epic
  • Release v1.0 of Meltano SDK - Epic

# December 2021

  • Documentation Overhaul - Epic
  • Monitoring, observability, and data lineage - Epic
  • Add Dagster as an Orchestrator - Epic & Issue

# 2022-H1

Next year will be continuing to make Meltano the best data integration tool on the market and Singer the best open source ecosystem. With this great foundation in place, we'll continue to invest in making Meltano the best DataOps platform on the market. This means focusing more on the user interface and on sell Meltano to paying customers.

# Contributing

Meltano is built for and by its community, and we welcome your contributions to our GitLab repository, which houses Meltano's core, CLI, UI, UI API, these docs, and the index of discoverable plugins, which feeds the lists of Extractors and Loaders that are supported out of the box.

To learn more about contributing to Meltano, refer to the Contributor Guide.

# History

Meltano was founded inside GitLab in 2018 to serve the GitLab Data Team. Meltano started as an open source tool built for GitLabโ€™s data and analytics team, who wanted an end-to-end data platform built around open source components and DevOps principles.

In 2020, the project strategically pivotedto focus on open source data integration (ELT).

In 2021, After 3 years of being a GitLab incubator project, Meltano spun out of GitLab and raised a seed funding round to become an independent startup and bring the benefits of open source data integration and DataOps to a wider audience.

# Team

Meltano is built by an all-remote team of 10 and a community of contributors. If you'd like to join the team, check out the career opportunities below!

# Values

Meltano's core values are community, empathy, sustainability, accountability, transparency, and iteration.

Our values act as a guide in how we conduct day-to-day business, make decisions, hire, and build our product. While we use these values as our compass, we adjust them based on lessons we've learned and our experiences using them in practice. Anyone can suggest improvements to our values - from team members to internet friends - and we're always open to growing and learning.

# Community

We work to build a more diverse and inclusive community by hiring globally and creating a culture that supports team members everywhere. We want to enable everyone to contribute - from team members to users - and we recognize that our vibrant community is at the core of Meltano. We will always keep the interests of our open source users and contributors in mind as we continue to grow.

# Empathy

We work with empathy, recognizing we're not alone in this journey. We bring kindness, humility, trust, honesty, and an assumption of good faith to all of our interactions.

# Sustainability

We work with a sustainable mindset so that we can continue to grow for years to come. We believe in investing in our people, community, and products for the long term. We do this by putting family and friends first, by being wise with resources, and by not adopting a short-term point of view. Sustainability plays a role in all aspects of our business - from hiring and developing to selecting swag options. We should strive to be a net positive company that makes net positive actions. Every change should be towards making "the thing" a little bit better than the one that came before.

# Accountability

We have a responsibility to ourselves, our teammates, and our community to do our best. We hold ourselves and each other to high standards, and we're responsible for our work, words, and actions. We're accountable for our mistakes, but we focus on learning from them and move forward together.

# Transparency

Transparency cultivates honesty which, in turn, builds trust. Without trust, it's impossible to live our other values. In practice, transparency can be seen in how we discuss decisions, document our work, and encourage everyone to share their thoughts and provide feedback. We work transparently, because we're passionate about involving our community in everything we're doing. We strive to provide users with the same level of insight as our team members so that they're able to have full visibility in our work and contribute their perspectives, since anything we build will be better if we collaborate with the people who will use it.

# Iteration

We work in small iterations so that we can be efficient, add value quickly, and get feedback from the community. We have a bias for action and aim to continually improve our product and processes, and we believe small, positive steps are the best way to achieve large goals. In practice, we ship minimum viable changes, and we continually seek to improve ourselves.

# Careers

The team is growing: we're planning to bring on 4 more people in the near future! See a complete list of our job openings!

If our mission excites you, and you think could make a great addition to the team in one of the following roles, we'd love to talk to you. Please send a cover letter and resume to hello@meltano.com and you'll hear back from us shortly.

# Employee Perks

  • Unlimited PTO
  • Professional Development Support and Budget
  • Remote Working
  • Reimbursable coworking fees and external office space
  • Budget for Office equipment and workspace supplies
  • Team Offsites
  • Access to world class founders, investors and mentors