Building Meltano in Public: Bimonthly Recap

Last week, it was once again my turn to host a GitLab Group Conversation (a publicly live streamed Q&A on the GitLab Unfiltered YouTube channel) on Meltano!

I used the opportunity to share a recap of:

• the 7 releases we’ve had since the last update 2 months ago,
• 17 recent contributions made by 11 community members,
• recent weekly Slack activity,
• other exciting recent and ongoing developments (including the brand-new Singer SDK and plans to hire community contributors onto the team at GitLab),
• this week’s priorities, and
• a list of upcoming priorities.

If you’re curious, check out the presentation on Google Slides and the Q&A on YouTube. The presentation content is also reproduced below, as is an embedded video of the Q&A!

Group Conversation Presentation

7 releases since the last GC (2020-11-19)

1. V1.59.0 makes sure that all meltano elt errors properly make it into the log file and UI and that meltano select --list prints entities and attributes in alphabetical order.
2. V1.60.0 adopts Poetry for dependency and build management and hides settings of kind object or array in the UI.
3. V1.61.0 adds out-of-the-box support for the BigQuery source using the tap-bigquery extractor, adds a new meltano schedule run <name> command to easily run a scheduled pipeline by name, shows array and object settings in the plugin configuration UI as unsupported, and fixes the “Run Now” button in the pipelines UI to take into account the schedule’s overridden environment variables.
4. V1.62.0 introduces plugin inheritance to let you have multiple configurations of the same package in your project at the same time, in the form of separate plugins that inherit their base plugin (package) description from an existing plugin and can then override (parts of) the inherited configuration.
5. V1.63.0 automatically retries failed connections to the system database, and lets you tweak this behavior using new database_max_retries (default: 3) and database_retry_timeout (default: 5 seconds) settings.
6. V1.64.0 fixes runaway memory consumption when an extractor outputs records at a much higher rate than the loader can process them, by enabling flow control with a 64KB buffer size limit. 
7. V1.65.0 lets you tweak the size of the buffer that stores records output by the extractor (tap) while they are waiting to be processed by the loader (target), using a new elt.buffer_size setting.

(Earlier today, we also released V1.66.0!)

17 recent contributions by 11 community members

Done

1. Enable `pool_pre_ping` for the project’s DB engine by Suyash Behera (Goldman Sachs)
2. Update the ELT runner to explicitly log any error messages by Allan Whatmough (Run with AI)
3. Added sorting for `meltano select –list –all` by Nil
4. Added missing `mysql-logo.png` by Nil
5. Adding Poetry for better dependency and build management by Tobias Macey (MIT)
6. Make tap bigquery known to meltano by Niall Woodward (Tails.com)
7. Adding pre-commit and linting configurations by Tobias Macey (MIT)
8. Tap-salesforce:  Pass is_sandbox to authenticator by Kevin Ford
9. Files-dbt: Add bigquery profile by Daniel Pettersson (Smartr)
10. Linting fixes and contributor guide update by Tobias Macey (MIT)
11. Target-postgres: Fix non-null falsey values by Charles Julian Knight (FIXD)

In development

1. Search and replace “entities” >> “streams”, “attributes” >> “properties” by AJ Steers (Slalom)
2. Add pipx-based install instructions by AJ Steers (Slalom)
3. Singer SDK: Accelerated tap development framework (v0.0.1-alpha) by AJ Steers (Slalom)
4. Singer SDK: Initial mock-up for target-base by AJ Steers (Slalom)
5. Added libpq required target-postgres dependency install instructions by Geoff Langenderfer
6. Target-snowflake: Dynamic precision fix by Bryan Wise (Halosight)

Recent weekly Slack activity

As expected, activity dropped significantly over the holidays, but it’s steadily climbing back to our previous records of 157 “weekly active members” (Dec 9) and 30 “members who posted” (Nov 18).

Join us on Slack!

Group Conversation Q&A