Data Integration and Data Observability go hand in hand. That’s why the folks at Databand, creators of the first proactive data observability platform, invited me to join them on the Databand Podcast to talk about how Meltano began, where it is now, and what’s in store for the future. Here’s a recap of what we discussed. If you’re interested in listening to the entire conversation, you can here.
The Vision of Meltano
The original vision of Meltano, when it was incubated inside GitLab, was to build an end-to-end solution for data teams. As the product and community grew, Meltano was able to spin out of GitLab with an even larger mission and vision: enable everyone to realize the full potential of their data by becoming the foundation of every team’s ideal data stack. To achieve this ambitious vision, Meltano is building a DataOps OS that brings the best practices of software engineering to data teams everywhere.
An example of bringing DevOps best practices to the data stack is Meltano’s new environments and tests feature. Environments enable users to define unlimited configuration bundles for development versus production deployments. The Meltano test command enables any plugin to run its test all with a single command. Meltano Environments plus Tests make it easy to spin up a staging environment, run the data integration and end-to-end tests, and once everything looks good, deploy to production.
As the Meltano product and community grows we aim to continue serving teams of all sizes. Meltano currently integrates with the Singer project for data replication, dbt for transformation, and Airflow for orchestration, with plans to add many more plugins such as Great Expectations, Dagster, Superset, and more. Our focus on open source tools and open core vendors will enable Meltano to bring high quality tooling to teams of all sizes.
Why Open Source
It is a core belief at Meltano that open source and open core tools can compete with the biggest and the best. We truly believe that for any commercial product out there, there will be some open source competitors that are as good or better. A particular example of this is data integration. Most data integration vendors typically top out around 150 to 200 connectors. Open source unlocks the cap on the number of connectors by specifying the data transfer format so that any number of connectors can be built to the specification.
Meltano uses the open source Singer project for data extraction and loading. Singer specifies how data should be structured so that any number of extractors, also called taps, can be used with any of the loaders, also called targets. By choosing open source, it makes it possible for teams of any size to build and use connectors for data sources that may not be supported by paid vendors.
However, there are few challenges with having such a large number of connectors. Simply finding them can be difficult and understanding their overall quality might not be easy either. This is why MeltanoHub was created. It’s a place to discover and share all the Singer connectors, and other Meltano plugins, that the data community is building to address their data needs.
The Future of Meltano
The broader vision for Meltano is to build a DataOps OS for the entire data lifecycle. There’s a need for an overarching operating system in the current data ecosystem. Fragmentation in the data stack is a pain point for data teams, and companies are buying a variety of solutions as a product evolves to help solve for this fragmentation. Meltano provides a data operations foundational layer to enable bilateral integrations between individual tools in the modern stack. End-to-end testing will be available between isolated environments, and users will have an high level overview of their entire data stack when this broader vision is fully realized.
In the next year, Meltano is aimed at finding true product-market fit with this DataOps OS vision. With data integration as a core capability, Meltano aims to help data engineers solve real problems while building the future of data tooling that unlocks the potential of data for everyone. Meltano has investors that believe in this mission, and it has a dedicated community that is excited about what’s being done. By building Meltano as an open source project in close collaboration with the community will enable us to grow and earn trust, usage, and contributions will help us achieve the longer term vision.