How we Built Multi-file Support with the Meltano Community

One of the best parts of being an open source project with a strong supporting community is collaboration. Our community is fantastic and many of our best contributions come from community members. 

A recent example of this is a new feature that we launched with v1.87.0 of Meltano: support for multi-file `meltano.yml` configurations. Now, instead of editing one large file, many smaller files can be used to scope plugins, schedules, and environments to specific teams or even users.

Origin Story of Multi-file meltano.yml configuration

The original issue for this idea came from Douwe Maan, our CEO, almost exactly 1 year ago. Back then, Douwe was the only person working on the project full time as an employee of GitLab. Like we do with many of our ideas, he created an issue to centralize discussion of the feature request. 

Over the coming months, a lot happened with Meltano as a project and company – we spun out of GitLab, launched MeltanoHub and Meltano SDK, and grew the team. But this particular issue wasn’t a priority as it’s generally a “nice-to-have” rather than something that will drive adoption. 

Eventually, though, Matthew Thurman, a Meltano community member, left a comment saying he was willing to contribute this feature as it would help his team at Gusto to better collaborate on the project. 

Co-Development

From there, the conversation happened asynchronously in the issue (and a bit on Slack) about how implementation would happen. Matthew was able to get a merge request in and the team started to provide feedback where needed.

One of the fun parts of building software is that things that are easy to spec can be quite challenging to implement! There was a lot of discussions in the issue and MR about the best way to add this feature. Eventually, we decided to implement it in a separate MR. We felt the changes we were requesting on Matthew’s original MR were too significant to expect a community member to fully implement. Ken Payne, our most recent backend engineer hire, took the work that Matthew started across the finish line.

In the wild

With this new capability, we’re excited that a better and more collaborative experience is available to all Meltano users. This should make it much easier to organize large projects.

Of course we want to keep iterating and improving on this and other features. We opened this issue to refactor multi-file configuration to enable first-class performance and compatibility. Enabling this will unlock other exciting features like preserving comments and flow styles when programmatically editing the meltano.yml file (issue link).

Build with Us

Community is one of our core values, and hopefully this example shows you how we live that value. Meltano is a product built by and for its community. If you have ideas or code to contribute on how to improve it, please let us know! We’d love to work with you to make your ideas come to life.

Join us in Slack or come to our weekly Office Hours to share how it’s going. Additionally, you can file an issue if you find a bug. Thanks for being a part of this journey!

Intrigued?

You haven’t seen nothing yet!

Join our mailing list

Stay current with all things Meltano