Building Meltano in Public: Bimonthly Recap

Last week, it was once again our turn to host a GitLab Group Conversation (a publicly live streamed Q&A on the GitLab Unfiltered YouTube channel) on Meltano!

We used the opportunity to share a recap of:

• the 2 new team members who just joined: Taylor Murphy and AJ Steers,
• the 5 releases we’ve had since the last update 2 months ago,
• 17 recent contributions made by 9 community members,
• other exciting recent and ongoing developments,
• recent and projected weekly Slack activity,
• that week’s priorities, and
• a list of upcoming priorities.

If you’re curious, check out the presentation on Google Slides and the Q&A on YouTube. The presentation content is also reproduced below, as is an embedded video of the Q&A!

Group Conversation Presentation

2 new team members just joined!

1. Taylor Murphy joined on March 1
   • Taylor comes from the GitLab Data team, where he was Staff Data Engineer responsible for architecture
   • Announcement blog post: https://meltano.com/blog/2021/03/01/joining-the-team-taylor-murphy-shares-why-hes-excited/

2. AJ Steers joined on March 8
   • AJ comes from Slalom, where he was a Solution Principal building next-gen products and reusable frameworks to solve today’s hardest data problems
   • As a contributor, he initiated the Singer SDK project: https://gitlab.com/meltano/singer-sdk
   • Announcement blog post: WIP

With Douwe Maan, that makes 3 full-time Meltano team members!

5 releases since the last GC (2021-01-14)

1. V1.66.0 prevents pipelines from getting stuck in the “running” state forever when their meltano elt process is killed unceremoniously by the operating system or some other mechanism.

2. V1.67.0 fixes two bugs with meltano schedule run <name>: if the schedule’s meltano elt command fails with a nonzero exit code, it now does as well, and it no longer requires the meltano executable to be in the PATH.

3. V1.68.0 adds support for entity/attribute selection to tap-gitlab and bumps Airflow to version 1.10.14.

4. V1.69.0 adds out-of-the-box support for the Quickbooks source and adds support for Airflow 2.

5. V1.70.0 prevents two pipelines with the same Job ID from running at the same time, adds out-of-the-box support for the Chargebee and Sage Intacct sources, and fixes a bug that prevented Meltano Analyze from working with custom plugins.

17 recent contributions by 9 community members

Done

1. target-snowflake: Dynamic Precision Fix by Bryan Wise (Halosight)
2. tap-gitlab:  Implement discovery mode and stream selection by Charles Julian Knight (FIXD)
3. Add support for Airflow 2.0″ by Michel Radosavljevic (Insurello)
4. Add `hotgluexyz` variant of `tap-quickbooks` by Hassan Syyid (Hotglue)
5. Document that tap-postgres depends on libpq by Derek Visch (Auto IDM)
6. Docs Development Steps NPM run serve:docs Missing Script by Derek Visch (Auto IDM)
7. Add `hotgluexyz` variant of `tap-chargebee` by Hassan Syyid (Hotglue)
8. Fix Meltano Analyze with custom extractor by Al Whatmough
9. Add `hotgluexyz` variant of `tap-intacct` by Hassan Syyid (Hotglue)
10.  Refactor Job sync implementation by Charles Julian Knight (FIXD)
11. files-airflow:  Add pipeline details into tags by Pankaj Saini (Let’s Do This)

In development

1. Support for a generic plugin type for arbitrary python cli plugins by Charles Julian Knight (FIXD)
2. Add support for shortcut commands for plugins by Charles Julian Knight (FIXD)
3. Adding pipelinewise target-redshift by Ingo Klose (b.telligent)
4. Disallow parallel pipelines by default by Charles Julian Knight (FIXD)
5. Add tap-zendesk variant by Benjamin Maquet (Twilio)
6. singer-sdk: Fix reading catalog from JSON file by Edgar R. Mondragón (SpotOn)

Other exciting recent and ongoing developments

• 20 community members created 33 new issues

• Meltano reached #1 on r/dataengineering: https://www.reddit.com/r/dataengineering/comments/l863az/meltano_elt_for_the_devops_era_open_source/

• Meltano got 111 upvotes on Hacker News: https://news.ycombinator.com/item?id=26284253

• Andrew Stewart wrote a blog post about Meltano + dbt + Superset: https://towardsdatascience.com/data-stacks-for-fun-nonprofit-part-iii-dcfd46da9f9f

• John Timeus, Derek Visch, and Edgar R. Mondragón have started putting the upcoming Singer SDK to the test with brand-new taps for PowerBI, BambooHR, and Confluence!

Recent and projected weekly Slack activity

Join 850+ on Slack!

This week’s priorities

Milestone issue board

• Onboarding
  • Taylor Murphy
  • AJ Steers
• Documentation
  • Add page listing maintainers of discoverable extractors and loaders
  • Document process of adopting abandoned taps/targets
  • Add new doc for `dbt` plugin and its settings
  • Add new doc for `airflow` plugin and its settings

Upcoming priorities

Milestones issue board

Epics:

• Building and maintaining Singer taps and targets
• Library of discoverable extractors and loaders
• Improved dbt integration
• ETL using Python scripts or YAML-defined rules
• Integrate with more (types of) plugins
• Debugging pipelines

Group Conversation Q&A