Building Meltano in Public: Bimonthly Recap

Last week, it was once again our turn to host a GitLab Group Conversation (a publicly live streamed Q&A on the GitLab Unfiltered YouTube channel) on Meltano!

We used the opportunity to share a recap of:

If you’re curious, check out the presentation on Google Slides and the Q&A on YouTube. The presentation content is also reproduced below, as is an embedded video of the Q&A!

Group Conversation Presentation

4 releases since the last GC (2021-03-15)

  1. V1.71.0Adds support for utility plugin type, added out of the box support for tap-zoom, fixes a bug to prevent false positive matches in passwords 
  1. V1.72.0 –  Added support for shortcut commands to invoke,  Added support for sqlfluff utility for linting SQL transforms,  Added mashey variant of tap-slack,  Added documentation for using a custom Python Package Index (PyPi),  Added transferwise variant of target-redshift. Fixes an error when when target process fails before tap. 
  1. V1.73.0 – Added twilio-labs variant of tap-zendesk, Speed up meltano install by installing plugins in parallel, Added support for setting kind in settings prompt when using meltano add –custom
  1. V1.74.0 – Add meltano remove command

SDK for Singer Taps Launched with 5 Subsequent Releases!

  1. v0.1.0 – SDK Launched! Announcement post: https://meltano.com/blog/2021/04/05/meltano-launches-v0-1-0-of-the-singer-tap-sdk/ 
  1. v0.1.1 –  Added ‘admin_name’ field in cookiecutter, streamline poetry setup, Added meltano integration and testing options , Added new cookiecutter .sh script to ease testing during development
  1. v0.1.2 – Fixes bug in Stream.get_starting_timestamp() using incorrect state key
  1. v0.1.3Large update to add support for unsorted streams
  1. v0.1.4Adds support for GraphQL variables and easy stream selection
  2. v0.1.5 – Resolved tap failure when a sorted stream has non-unique replication keys

22 Recent contributions by 9 community members

Done

  1. Meltano: Support for a generic plugin type for arbitrary python cli plugins by Charles Julian Knight (FIXD)
  2. Meltano: Add documentation for using a custom PyPI package by Benjamin Maquet (Twilio)
  3. Meltano: Updated the database_uri documentation for target-postgres by Daniel Walker (Matatika)
  4. Meltano: Adding pipelinewise target-redshift by Ingo Klose (b.telligent)
  5. Meltano: Add support for shortcut commands for plugins by Charles Julian Knight (FIXD)   
  6. Meltano: Disallow parallel pipelines by default by Charles Julian Knight (FIXD)
  7. Meltano: Add tap-zendesk variant by Benjamin Maquet (Twilio)
  8. Meltano: Configure kind of setting via prompt when adding a custom plugin by Reuben Frankel (Matatika)
  9. Meltano: Add `meltano remove <type> <name>` command to remove a plugin  by Reuben Frankel (Matatika)
  10. Singer-sdk: Fix reading catalog from JSON file by Edgar R. Mondragón (SpotOn)
  11. Singer-sdk: Fixed bug in STATE handling by Ken Payne (Tails.com)
  12. Singer-sdk: Look only for valid plugin settings in environment variables by Edgar R. Mondragón (SpotOn)
  13. Singer-sdk: Fixed issue reading from JSON file by Edgar R. Mondragón (SpotOn)
  14. Singer-sdk: Added default value support for plugin configs by Ken Payne (Tails.com)
  15. Singer-sdk: Fixed a problem where CI pipelines would fail when run from a fork by Derek Visch (Auto IDM)
  16. Singer-sdk: Fixed a scenario where unsorted incremental streams would generate incorrect STATE bookmarks by Egi Gjevori (y42)

In development

  1. Meltano: Update default dbt version to 0.18.2 by Andrew Stewart
  2. Meltano: Add fork of target-csv as new default variant by Hassan Syyid (Hotglue)
  3. Singer-sdk:Add built-in support for FAST_SYNC spec by Ken Payne (Tails.com)
  4. Meltano: Auto-generate dbt sources.yml from extractor/loader schema by Andrew Stewart
  5. Singer-sdk: Support defining configuration and stream schemas using Pydantic by Edgar R. Mondragón (SpotOn)
  6. Meltano: Allow Meltano to be run from subdirectories by Charles Julian Knight (FIXD)

Other exciting recent and ongoing developments

Recent and projected weekly Slack activity

This week’s priorities

Upcoming priorities

Milestones issue board

Epics:

Group Conversation Q&A