Recently, the Core Team met to discuss our visions of what Meltano could become over the coming years. It was a wonderful, multi-hour discussion and all of us left feeling excited about the potential and eager to manifest that future. In this post we wanted to share the highlights of what we talked about and how we think about the future of Meltano.
Constant Commitment to Open Source and the Data Community
While we have a lot of features on the roadmap, one of the things we talked about is the importance of our values and the community. To that end, we recognized that having a healthy open source ecosystem and a strong data community are going to be key drivers of our success over the long term.
One part of our mission with Meltano is to level-up the Data Profession. This means recognizing the value and competitive advantage that Data Professionals and Data Teams bring to an organization. Many Data Teams were built as an afterthought when an organization realized it “needed to do something with its data”. The state of tooling for Data Teams reflects that reality: it is disjointed, challenging to use, and squanders a huge opportunity for organizations everywhere.
We believe that building open-source software and fostering and supporting OSS within the data ecosystem is essential to achieving our full mission. OSS has elevated the best practices within software development and we know it can have the same effect on the Data Profession. We believe Singer will be the de facto standard for any data integration challenge. We also believe that Open Source will win many parts of the data stack. We want to integrate with and support those tools as well.
We also want the Meltano community to support all data professionals, not just Data Engineers. We aim to provide training and education for the entire data lifecycle, including for Meltano specifically. As the data profession grows into fully embracing DataOps we want to help people onboard to this new and better way of working. With so many open source tools available it can be daunting to know where or how to get started. We aim to make the tooling as understandable as possible while giving people the confidence to jump in and get working with a thoughtful, curated stack that supports modern best practices.
We recognize the fantastic responsibility of stewarding an open source project and community. We aim to do what’s best for Meltano and the larger data community so that we can sustainably build a great tool for years to come.
In 5 years – Be the One-Stop Shop for Data Teams
In the long-term we aim to go big. We believe that Meltano can become the one-stop shop for the majority of data projects. Here are some of the key things we want to see in Meltano less than 5 years from now.
Deep First Principles Integration – Issue #2667
We believe it will be critical to have DevOps best practices in the DNA of the product. This means having a strong notion of version control, CI/CD, and the development experience. It should be easy to have disposable environments for the entire stack (think about Snowflake zero copy clones). We believe these best practices to be important because DevOps tooling enables software teams to move faster with more confidence, reliability, and security – Data Teams will need to move faster as well since high-performing Data Teams are becoming more critical to the success of all businesses.
Fantastic User Interface – Epic #78
While we are building Meltano CLI-first, we know it will be essential to have a great UI for the tool as well. We believe that building for the command line first and growing from there will lead to a better experience and growth path for the product. But for us to be able to compete and bring in a whole new generation of data professionals we will need an excellent user experience with a well designed UI.
Enterprise Ready – Issue #2667
We want Meltano to be a long-lived project with a sustainable business supporting it. To do that we need to sell to large enterprises. This means thinking about features needed by these organizations before we’re ready to sell to them. We believe we can do this in a way that is supportive of the larger data community by following a buyer-based open core business model where only those features that are more relevant for managers and up are what we charge for. We will also want to have a SaaS and a Self-Hosted option that are equivalent to each other.
Go Big!
Meltano has a big future ahead of it and we’re confident in our ability to manifest it. But how do we get there? One iteration at a time. To that end, we also talked about where we want to be within the next year.
Meltano by 2022-H1
By next year we want to have all of the following done (and more, of course!).
Modernized and Enhanced Singer Ecosystem – Epics #87, #85, #80, & #88
Our work with the Singer Software Development Kit (SDK) is well underway and it will become the de facto way to build new taps and targets. Along with the SDK, we will soon be building the SingerHub which will be the easiest way to find working and up-to-date taps and targets.
These investments into the Singer community will continue with support and enhancement of the protocol itself. We’ve already released a first iteration of our interpretation of the Singer spec. We also plan to update and support optional extensions such as `ACTIVATE_VERSION` and `BATCH` (aka Fast Sync) message types. We’re also thinking about how to have a generic data plugin architecture for Singer so that data can be obfuscated or dropped on the fly.
Enhance the Command Line and Configuration Experience
Because we’re building Meltano with a CLI-first focus, we want to make working on the command line within your Meltano project even better. Bringing better decentralization and composability to a project is a part of that goal. This means making it easier to simplify configuration and have it spread out across your Meltano project by being able to split up your `meltano.yml` file and even import other projects into one another. Since a key part of the promise of Meltano is that it will integrate with many tools, being able to integrate with itself will be a core concept. Making Meltano git-aware is something we’re eager to add as well since it will help with the development and deployment of Meltano.
Metadata, Monitoring, Observability – Epic #93
Meltano as a product sits in a unique position because it is at the start of the lifecyle for any data project. Part of the promise of an all-in-one tool is that it makes it possible to easily understand that status of data anywhere along the value chain. We want to make metadata a native part of the product and build this capability early.
Great Deployment Experience – Epic #28
Deploying Meltano is a critical part of making it usable. We want to make it as easy as possible to deploy Meltano in any environment. To do this we’re going to enhance our dogfooding of the product as well as simplify the deployment process as much as possible. Great documentation will be a key piece of this too.
Full Open Source End-to-End Solution
Out of the box we want to have everything you’d need to get a modern data stack up and running. This means everything from data integration to orchestration, warehousing, transformation, operational analytics (Reverse ETL, if you like), and reporting. We’re going to do this by integrating with tools like Dagster, Prefect, Superset, and Jupyter Notebooks, but also by having support for a modern analytic datastore such as Spark, Clickhouse, or DuckDB. We’re also going to dramatically improve our dbt integration by doing things like autogenerating sources.yml files and adding native support for testing. Having this end-to-end stack in one place will enable unprecedented coordination between the different tools meaning we’ll be in a great place to achieve our long term vision.
The Future is Bright
As you can see, we’re all extremely excited about this space. We want to build a tool for Data Professionals so that organizations can manage and access their data more effectively so they can enable better decisions. We want to level-up the status of the Data Profession so that those who work in it are respected for the true value they can bring. We believe that the data community is growing more technical while the value of data within organizations is becoming more valuable.
We hope to support the community through the Meltano product but also via the open source ecosystem too. Meltano will be driven by what the community wants and needs, so we need your active involvement! Please collaborate with us on issues, in Slack, and on Twitter for how we can make the product and the community better!