Building Meltano in Public: Bimonthly Recap
Last week, it was once again our turn to host a GitLab Group Conversation (a publicly live streamed Q&A on the GitLab Unfiltered YouTube channel) on Meltano!
We used the opportunity to share a recap of:
- the 4 Meltano releases we’ve had since the last update 2 months ago,
- the SDK for Singer Taps launch and 5 subsequent releases
- 22 recent contributions made by 9 community members,
- other exciting recent and ongoing developments,
- recent and projected weekly Slack activity,
- that week’s priorities, and
- a list of upcoming priorities.
If you’re curious, check out the presentation on Google Slides and the Q&A on YouTube. The presentation content is also reproduced below, as is an embedded video of the Q&A!
Table of Contents
Group Conversation Presentation
4 releases since the last GC (2021-03-15)
- V1.71.0 – Adds support for utility plugin type, added out of the box support for tap-zoom, fixes a bug to prevent false positive matches in passwords
- V1.72.0 – Added support for shortcut commands to invoke, Added support for sqlfluff utility for linting SQL transforms, Added mashey variant of tap-slack, Added documentation for using a custom Python Package Index (PyPi), Added transferwise variant of target-redshift. Fixes an error when when target process fails before tap.
- V1.73.0 – Added twilio-labs variant of tap-zendesk, Speed up meltano install by installing plugins in parallel, Added support for setting kind in settings prompt when using meltano add –custom
- V1.74.0 – Add meltano remove command
SDK for Singer Taps Launched with 5 Subsequent Releases!
- v0.1.0 – SDK Launched! Announcement post: https://meltano.com/blog/2021/04/05/meltano-launches-v0-1-0-of-the-singer-tap-sdk/
- v0.1.1 – Added ‘admin_name’ field in cookiecutter, streamline poetry setup, Added meltano integration and testing options , Added new cookiecutter .sh script to ease testing during development
- v0.1.2 – Fixes bug in Stream.get_starting_timestamp() using incorrect state key
- v0.1.4 – Adds support for GraphQL variables and easy stream selection
- v0.1.5 – Resolved tap failure when a sorted stream has non-unique replication keys
22 Recent contributions by 9 community members
Done
- Meltano: Support for a generic plugin type for arbitrary python cli plugins by Charles Julian Knight (FIXD)
- Meltano: Add documentation for using a custom PyPI package by Benjamin Maquet (Twilio)
- Meltano: Updated the database_uri documentation for target-postgres by Daniel Walker (Matatika)
- Meltano: Adding pipelinewise target-redshift by Ingo Klose (b.telligent)
- Meltano: Add support for shortcut commands for plugins by Charles Julian Knight (FIXD)
- Meltano: Disallow parallel pipelines by default by Charles Julian Knight (FIXD)
- Meltano: Add tap-zendesk variant by Benjamin Maquet (Twilio)
- Meltano: Configure kind of setting via prompt when adding a custom plugin by Reuben Frankel (Matatika)
- Meltano: Add `meltano remove <type> <name>` command to remove a plugin by Reuben Frankel (Matatika)
- Singer-sdk: Fix reading catalog from JSON file by Edgar R. Mondragón (SpotOn)
- Singer-sdk: Fixed bug in STATE handling by Ken Payne (Tails.com)
- Singer-sdk: Look only for valid plugin settings in environment variables by Edgar R. Mondragón (SpotOn)
- Singer-sdk: Fixed issue reading from JSON file by Edgar R. Mondragón (SpotOn)
- Singer-sdk: Added default value support for plugin configs by Ken Payne (Tails.com)
- Singer-sdk: Fixed a problem where CI pipelines would fail when run from a fork by Derek Visch (Auto IDM)
- Singer-sdk: Fixed a scenario where unsorted incremental streams would generate incorrect STATE bookmarks by Egi Gjevori (y42)
In development
- Meltano: Update default dbt version to 0.18.2 by Andrew Stewart
- Meltano: Add fork of target-csv as new default variant by Hassan Syyid (Hotglue)
- Singer-sdk:Add built-in support for FAST_SYNC spec by Ken Payne (Tails.com)
- Meltano: Auto-generate dbt sources.yml from extractor/loader schema by Andrew Stewart
- Singer-sdk: Support defining configuration and stream schemas using Pydantic by Edgar R. Mondragón (SpotOn)
- Meltano: Allow Meltano to be run from subdirectories by Charles Julian Knight (FIXD)
Other exciting recent and ongoing developments
- Hosted 12 Community meetings (9 Office Hours (YouTube) and 3 Demo Days (YouTube))
- 1000+ Slack Members! Announcement Post: https://meltano.com/blog/2021/04/23/community-milestone-1000-slack-members/
- Meltano Vision blog post released https://meltano.com/blog/2021/04/09/our-vision-for-the-future-of-meltano/
- Released a Simplified interpretation of the Singer Specification https://meltano.com/blog/2021/04/07/a-simplified-take-on-the-singer-spec/
- Douwe did a 0 to ELT in 90 Seconds speedrun
- Reinvited to the Data Engineering Podcast
- Singer SDK featured in Tristan Handy’s Data Science Roundup
- Singerly.co launches – SaaS that is “Powered by Meltano” https://www.singerly.co/
- Comparative analysis of Meltano and other company’s roadmaps (GitHub, Atlassian, etc.) https://medium.com/geekculture/4-excellent-examples-of-agile-nearsighted-roadmaps-ffec9cf6de5e#dc01-cad406eb5cfb
- Prominent analysis in the Preset blog https://preset.io/blog/2021-4-22-data-integration-tooling/
Recent and projected weekly Slack activity
This week’s priorities
Upcoming priorities
Epics:
- Building and maintaining Singer taps and targets
- MeltanoHub Hard Launch
- Improved dbt integration
- ETL using Python scripts or YAML-defined rules
- Integrate with more (types of) plugins
- Debugging pipelines
Group Conversation Q&A
Category:
Community, Month in Review
