Building Meltano in Public: Bimonthly Recap

Last week, it was once again our turn to host a GitLab Group Conversation (a publicly live streamed Q&A on the GitLab Unfiltered YouTube channel) on Meltano!

We used the opportunity to share a recap of:

• the 4 Meltano releases we’ve had since the last update 2 months ago,
• the SDK for Singer Taps launch and 5 subsequent releases
• 22 recent contributions made by 9 community members,
• other exciting recent and ongoing developments,
• recent and projected weekly Slack activity,
• that week’s priorities, and
• a list of upcoming priorities.

If you’re curious, check out the presentation on Google Slides and the Q&A on YouTube. The presentation content is also reproduced below, as is an embedded video of the Q&A!

Group Conversation Presentation

4 releases since the last GC (2021-03-15)

1. V1.71.0 – Adds support for utility plugin type, added out of the box support for tap-zoom, fixes a bug to prevent false positive matches in passwords 

2. V1.72.0 –  Added support for shortcut commands to invoke,  Added support for sqlfluff utility for linting SQL transforms,  Added mashey variant of tap-slack,  Added documentation for using a custom Python Package Index (PyPi),  Added transferwise variant of target-redshift. Fixes an error when when target process fails before tap. 

3. V1.73.0 – Added twilio-labs variant of tap-zendesk, Speed up meltano install by installing plugins in parallel, Added support for setting kind in settings prompt when using meltano add –custom

4. V1.74.0 – Add meltano remove command

SDK for Singer Taps Launched with 5 Subsequent Releases!

1. v0.1.0 – SDK Launched! Announcement post: https://meltano.com/blog/2021/04/05/meltano-launches-v0-1-0-of-the-singer-tap-sdk/ 

2. v0.1.1 –  Added ‘admin_name’ field in cookiecutter, streamline poetry setup, Added meltano integration and testing options , Added new cookiecutter .sh script to ease testing during development

3. v0.1.2 – Fixes bug in Stream.get_starting_timestamp() using incorrect state key

4. v0.1.3 – Large update to add support for unsorted streams

5. v0.1.4 – Adds support for GraphQL variables and easy stream selection
6. v0.1.5 – Resolved tap failure when a sorted stream has non-unique replication keys

22 Recent contributions by 9 community members

Done

1. Meltano: Support for a generic plugin type for arbitrary python cli plugins by Charles Julian Knight (FIXD)
2. Meltano: Add documentation for using a custom PyPI package by Benjamin Maquet (Twilio)
3. Meltano: Updated the database_uri documentation for target-postgres by Daniel Walker (Matatika)
4. Meltano: Adding pipelinewise target-redshift by Ingo Klose (b.telligent)
5. Meltano: Add support for shortcut commands for plugins by Charles Julian Knight (FIXD)   
6. Meltano: Disallow parallel pipelines by default by Charles Julian Knight (FIXD)
7. Meltano: Add tap-zendesk variant by Benjamin Maquet (Twilio)
8. Meltano: Configure kind of setting via prompt when adding a custom plugin by Reuben Frankel (Matatika)
9. Meltano: Add `meltano remove <type> <name>` command to remove a plugin  by Reuben Frankel (Matatika)
10. Singer-sdk: Fix reading catalog from JSON file by Edgar R. Mondragón (SpotOn)
11. Singer-sdk: Fixed bug in STATE handling by Ken Payne (Tails.com)
12. Singer-sdk: Look only for valid plugin settings in environment variables by Edgar R. Mondragón (SpotOn)
13. Singer-sdk: Fixed issue reading from JSON file by Edgar R. Mondragón (SpotOn)
14. Singer-sdk: Added default value support for plugin configs by Ken Payne (Tails.com)
15. Singer-sdk: Fixed a problem where CI pipelines would fail when run from a fork by Derek Visch (Auto IDM)
16. Singer-sdk: Fixed a scenario where unsorted incremental streams would generate incorrect STATE bookmarks by Egi Gjevori (y42)

In development

1. Meltano: Update default dbt version to 0.18.2 by Andrew Stewart
2. Meltano: Add fork of target-csv as new default variant by Hassan Syyid (Hotglue)
3. Singer-sdk:Add built-in support for FAST_SYNC spec by Ken Payne (Tails.com)
4. Meltano: Auto-generate dbt sources.yml from extractor/loader schema by Andrew Stewart
5. Singer-sdk: Support defining configuration and stream schemas using Pydantic by Edgar R. Mondragón (SpotOn)
6. Meltano: Allow Meltano to be run from subdirectories by Charles Julian Knight (FIXD)

Other exciting recent and ongoing developments

• Hosted 12 Community meetings (9 Office Hours (YouTube) and 3 Demo Days (YouTube))
• 1000+ Slack Members! Announcement Post: https://meltano.com/blog/2021/04/23/community-milestone-1000-slack-members/
• Meltano Vision blog post released https://meltano.com/blog/2021/04/09/our-vision-for-the-future-of-meltano/
• Released a Simplified interpretation of the Singer Specification https://meltano.com/blog/2021/04/07/a-simplified-take-on-the-singer-spec/
• Douwe did a 0 to ELT in 90 Seconds speedrun
• Reinvited to the Data Engineering Podcast
• Singer SDK featured in Tristan Handy’s Data Science Roundup
• Singerly.co launches – SaaS that is “Powered by Meltano” https://www.singerly.co/
• Comparative analysis of Meltano and other company’s roadmaps (GitHub, Atlassian, etc.) https://medium.com/geekculture/4-excellent-examples-of-agile-nearsighted-roadmaps-ffec9cf6de5e#dc01-cad406eb5cfb
• Prominent analysis in the Preset blog https://preset.io/blog/2021-4-22-data-integration-tooling/

Recent and projected weekly Slack activity

This week’s priorities

• Milestone issue board

• MeltanoHub 

• SDK

Upcoming priorities

Milestones issue board

Epics:

• Building and maintaining Singer taps and targets
• MeltanoHub Hard Launch
• Improved dbt integration
• ETL using Python scripts or YAML-defined rules
• Integrate with more (types of) plugins
• Debugging pipelines

Group Conversation Q&A