Building Meltano in Public: four weeks into the relaunch
Earlier this week, it was my turn to host a GitLab Group Conversation (a publicly live streamed Q&A on the GitLab Unfiltered YouTube channel) on Meltano. I used the opportunity to share some updates about last month’s announcement of Meltano’s new strategy and direction and the response and impact so far.
If you’re curious, check out the presentation on Google Slides and the Q&A on YouTube. The presentation content is also reproduced below, as is an embedded video of the Q&A.
Personally, I couldn’t be happier with how things have turned out so far, and I’m incredibly grateful to everyone who has given Meltano a try, shared feedback from their experience, and contributed to the future of this open source ELT solution.
In conclusion, allow me to quote from the Slack conversation I had with GitLab CEO Sid Sijbrandij after he saw the presentation:
- Sid: great content, love the quotes on slide 3, these people really get what you’re doing.
- Douwe: Yep, and attracting a few dozen people who really get it, will stick with it, and will help make it happen is exactly what I was aiming for, so I’d call this a successful relaunch!
Group Conversation Presentation
Getting the word out about Meltano’s new direction
In the previous Meltano GC, I shared drafts of 2 blog posts announcing Meltano’s new strategy and direction. These were published on the Meltano blog on May 13:
- Revisiting the Meltano strategy: a return to our roots
- Key phrase: We will go all-in on positioning Meltano as an open source self-hosted platform for running data integration and transformation (ELT) pipelines.
- Why we are building an open source platform for ELT pipelines
- Key phrase: Our goal is to make the power of data integration available to all by building a true open source alternative to existing proprietary hosted ELT solutions.
On May 18, I got a chance to talk about this new direction on the Drill to Detail podcast: ‘Meltano, Singer Taps and Open-Source Data Pipelines’ with Special Guest Douwe Maan
The goal: attract existing Singer community members, get them to give Meltano a try, gather feedback on how it currently stacks up as an ELT pipeline platform, and acquire contributors
The response has been overwhelmingly positive
Blog post 2 reached the frontpage of the r/dataengineering subreddit, with very positive responses: (emphasis mine)
- So if I understand it correctly the project is about building an open source community to build and maintain an array of SaaS APIs so people can spend less or no time getting that data, for free?
Seems brilliant and wonder why it hasn’t been done so far… Hell, just sorting out Facebook Ads and Google Campaign Manager APIs could be a game changer for a lot of smaller advertising agencies who otherwise can never afford the tools or expertise to do these in house.
Will be following closely and would love to maybe even try and contribute at some point!
- This is very exciting, I like the standing on the shoulders of giants approach, by sitting in top of existing leading open source initiatives in this space like airflow, dbt and singer, and making them easier and faster to coordinate and code with. I’ll be following this.
Responses on Twitter have been similarly positive:
- This sounds terrific! Really hope it can reinvigorate the Singer community… so far I’ve been really surprised by how much of a ghost town most taps are. This seems like it could be a great catalyst.
- I LOVE LOVE LOVE the new @meltanodata. This is what the future of the world’s data infrastructure looks like. If @stitch_data and @singer_io are the gateway drug… @meltanodata is the complete thing.
Impact by the numbers
- 3845 visitors to meltano.com between May 13, 14, and 15, where we normally would have seen about 100 per day (up 1182% from 300)
- Since the announcement:
- 55 new Twitter followers (up 4% from 1469)
- 64 new Slack workspace members (up 23% from 273)
- 351 new users of the CLI, where we normally would have seen about 170 (up 107%)
- 103 weekly active members on the Slack workspace 3 weeks after the announcement, up from 38 just before (up 171%)
- 6 new issues by and with comments from 6 people (excluding me) that have allowed me to flesh out and start working on the Running pipelines in production epic
- 5 new MRs by 3 people (excluding me)
- 6 new releases of Meltano with 15 bug fixes and many changes and new features, most in response to feedback from new users
Other positive outcomes
- I’ve been invited to talk about Meltano on The Data Engineering Podcast and will be interviewed on June 15
- RFA are considering replacing PipelineWise with Meltano in the AppliedData platform they offer to clients, and are planning to put 1.5 developers on Meltano for the next 2 months to work on the Data pipeline management UI epic
- Multiple other data engineering consultants and consultancies are evaluating Meltano and considering adopting it as part of the stack they set up for clients
- A commercial player in the hosted data pipeline orchestration space has expressed interest in finding ways to collaborate
- A few GitLab customers have informed about Meltano through their account managers
Epics for current priorities
- Running pipelines in production
- Encourage contribution
- Data pipeline management UI (in collaboration with Applied Labs!)