I recently had the opportunity to be a guest on the Hashmap on Tap Podcast to talk about Meltano and how we’re building a community around DataOps-driven ELT. It was a great conversation that explored Meltano’s approach to DataOps, our commitment to open source, and how Meltano is currently being used to help teams get the most from their data.
Software Development Approach to Data
DataOps and DevOps are different processes, undoubtedly, but the best practices that have been established in DevOps definitely translate to DataOps. In DevOps, a pipeline leads to the production and delivery of a product: a software application. Data stacks can be seen as products for their data teams who are working to gain the most from their data as efficiently as possible. In the case of DataOps, the users of this data stack product are the colleagues who use the dashboard. The task at hand for a data team is to continually improve and iteratively add capabilities to the data stack to serve those users better.
Using a software development approach to data addresses the way data teams can communicate and collaborate for the good of their organization. This is what we mean when we say Meltano is the DataOps OS. With Meltano, data teams get best practices like version control and code review, continuous integration and deployment, and isolated environments—all elements that are currently seen in a DevOps process. Having automated, end-to-end tests for every change made and having the ability to try things on your local machine without the chance of breaking things on the live dashboard gives data teams the confidence to approach their work with a spirit of experimentation and rapid iteration.
Open Source Benefits
Meltano is an open source tool, operating under the philosophy that the best tools are built in close collaboration with their users—in this case, data engineers. With skilled users having full insight into the product development process and giving you suggestions directly, tools will always be better, using all the ideas, perspectives, and code contributions of those using the tool to continually improve it. That’s what having an open source tool is all about; giving members of the community the best possible experience by making them integral to the journey. Each user is included in the decision making, prioritization, and functionality trade-offs.
Today, Meltano focuses primarily on data integration and ELT. Data engineers can expect to make changes directly in the Meltano YAML file or through the CLI while testing their data integration pipeline on their local machine. Going from zero to complete pipelines running in production in half a day or less is what they can count on. Incorporating Singer, the de-facto standard for open source data connectors, gives Meltano users access to a thriving community and ecosystem of connectors for more than 300 sources and destinations. This technology is a key component for ELT, along with dbt for transformation and Airflow for orchestration. Meltano has support for many data warehouses, and can be deployed on the cloud infrastructure of your choice.
DataOps with Meltano
There are several reasons why an organization would choose to go with Meltano. When considering an open source tool versus a proprietary data integration SaaS solution, it’s important to know that the library of connectors with open source is effectively endless. And having the ability to self-manage and self-host the software within your own infrastructure is the best way to deal with security and compliance requirements. When considering open source versus building your own Python scripts, the fact that you’re sharing code with a wider community lowers the maintenance burden. And, building a connector to Singer—the de-facto standard for open source data connectors—offers developers a great deal of functionality out of the box while reducing exposure to lock-in.
Today, Meltano is offering solutions to a diversity of users, including data engineers and software engineers that feel at home with Meltano when tasked with a data challenge. We’re finding that medium-sized tech companies with large engineering teams who can pick their own tools see a lot of value in Meltano, as do non-tech companies with small technical teams and data consultancies setting up data stacks for their clients.
In the future, we see Meltano becoming the foundation of an organization’s entire data stack. It will go beyond ELT to make DataOps a reality for every stage of the data lifecycle by bringing different open source data tools together in one place and allowing teams to engage with their stack as a single unit. Those who do not want to self-manage their data infrastructure will be able to host their entire stack on our managed Meltano platform. The core of the product will always remain open source and available for free.
We had a great conversation, and I encourage you to check out the podcast. Or, if you’re ready to explore the best open source ELT and the only DataOps OS, download Meltano and join our community today!