Building Meltano in Public: Bimonthly Recap
Last week, it was once again our turn to host a GitLab Group Conversation (a publicly live streamed Q&A on the GitLab Unfiltered YouTube channel) on Meltano!
We used the opportunity to share a recap of:
- the 2 new team members who just joined: Taylor Murphy and AJ Steers,
- the 5 releases we’ve had since the last update 2 months ago,
- 17 recent contributions made by 9 community members,
- other exciting recent and ongoing developments,
- recent and projected weekly Slack activity,
- that week’s priorities, and
- a list of upcoming priorities.
If you’re curious, check out the presentation on Google Slides and the Q&A on YouTube. The presentation content is also reproduced below, as is an embedded video of the Q&A!
Group Conversation Presentation
2 new team members just joined!
- Taylor Murphy joined on March 1
- Taylor comes from the GitLab Data team, where he was Staff Data Engineer responsible for architecture
- Announcement blog post: https://meltano.com/blog/2021/03/01/joining-the-team-taylor-murphy-shares-why-hes-excited/
- AJ Steers joined on March 8
- AJ comes from Slalom, where he was a Solution Principal building next-gen products and reusable frameworks to solve today’s hardest data problems
- As a contributor, he initiated the Singer SDK project: https://gitlab.com/meltano/singer-sdk
- Announcement blog post: WIP
With Douwe Maan, that makes 3 full-time Meltano team members!
5 releases since the last GC (2021-01-14)
- V1.66.0 prevents pipelines from getting stuck in the “running” state forever when their
meltano eltprocess is killed unceremoniously by the operating system or some other mechanism.
- V1.67.0 fixes two bugs with
meltano schedule run <name>: if the schedule’s
meltano eltcommand fails with a nonzero exit code, it now does as well, and it no longer requires the
meltanoexecutable to be in the
- V1.68.0 adds support for entity/attribute selection to tap-gitlab and bumps Airflow to version 1.10.14.
- V1.69.0 adds out-of-the-box support for the Quickbooks source and adds support for Airflow 2.
- V1.70.0 prevents two pipelines with the same Job ID from running at the same time, adds out-of-the-box support for the Chargebee and Sage Intacct sources, and fixes a bug that prevented Meltano Analyze from working with custom plugins.
17 recent contributions by 9 community members
- target-snowflake: Dynamic Precision Fix by Bryan Wise (Halosight)
- tap-gitlab: Implement discovery mode and stream selection by Charles Julian Knight (FIXD)
- Add support for Airflow 2.0″ by Michel Radosavljevic (Insurello)
- Add `hotgluexyz` variant of `tap-quickbooks` by Hassan Syyid (Hotglue)
- Document that tap-postgres depends on libpq by Derek Visch (Auto IDM)
- Docs Development Steps NPM run serve:docs Missing Script by Derek Visch (Auto IDM)
- Add `hotgluexyz` variant of `tap-chargebee` by Hassan Syyid (Hotglue)
- Fix Meltano Analyze with custom extractor by Al Whatmough
- Add `hotgluexyz` variant of `tap-intacct` by Hassan Syyid (Hotglue)
- Refactor Job sync implementation by Charles Julian Knight (FIXD)
- files-airflow: Add pipeline details into tags by Pankaj Saini (Let’s Do This)
- Support for a generic plugin type for arbitrary python cli plugins by Charles Julian Knight (FIXD)
- Add support for shortcut commands for plugins by Charles Julian Knight (FIXD)
- Adding pipelinewise target-redshift by Ingo Klose (b.telligent)
- Disallow parallel pipelines by default by Charles Julian Knight (FIXD)
- Add tap-zendesk variant by Benjamin Maquet (Twilio)
- singer-sdk: Fix reading catalog from JSON file by Edgar R. Mondragón (SpotOn)
Other exciting recent and ongoing developments
- 20 community members created 33 new issues
- Meltano reached #1 on r/dataengineering: https://www.reddit.com/r/dataengineering/comments/l863az/meltano_elt_for_the_devops_era_open_source/
- Meltano got 111 upvotes on Hacker News: https://news.ycombinator.com/item?id=26284253
- Andrew Stewart wrote a blog post about Meltano + dbt + Superset: https://towardsdatascience.com/data-stacks-for-fun-nonprofit-part-iii-dcfd46da9f9f
- John Timeus, Derek Visch, and Edgar R. Mondragón have started putting the upcoming Singer SDK to the test with brand-new taps for PowerBI, BambooHR, and Confluence!
Recent and projected weekly Slack activity
This week’s priorities
- Building and maintaining Singer taps and targets
- Library of discoverable extractors and loaders
- Improved dbt integration
- ETL using Python scripts or YAML-defined rules
- Integrate with more (types of) plugins
- Debugging pipelines
Group Conversation Q&A
Category: Month in Review