Building Meltano in Public: Bimonthly Recap

Last week, it was once again my turn to host a GitLab Group Conversation (a publicly live streamed Q&A on the GitLab Unfiltered YouTube channel) on Meltano!

I used the opportunity to share a recap of:

• the 7 releases we’ve had since the last update 2 months ago,
• 17 recent contributions made by 11 community members,
• recent weekly Slack activity,
• other exciting recent and ongoing developments (including the brand-new Singer SDK and plans to hire community contributors onto the team at GitLab),
• this week’s priorities, and
• a list of upcoming priorities.

If you’re curious, check out the presentation on Google Slides and the Q&A on YouTube. The presentation content is also reproduced below, as is an embedded video of the Q&A!

Group Conversation Presentation

7 releases since the last GC (2020-11-19)

1. V1.59.0 makes sure that all meltano elt errors properly make it into the log file and UI and that meltano select --list prints entities and attributes in alphabetical order.
2. V1.60.0 adopts Poetry for dependency and build management and hides settings of kind object or array in the UI.
3. V1.61.0 adds out-of-the-box support for the BigQuery source using the tap-bigquery extractor, adds a new meltano schedule run <name> command to easily run a scheduled pipeline by name, shows array and object settings in the plugin configuration UI as unsupported, and fixes the “Run Now” button in the pipelines UI to take into account the schedule’s overridden environment variables.
4. V1.62.0 introduces plugin inheritance to let you have multiple configurations of the same package in your project at the same time, in the form of separate plugins that inherit their base plugin (package) description from an existing plugin and can then override (parts of) the inherited configuration.
5. V1.63.0 automatically retries failed connections to the system database, and lets you tweak this behavior using new database_max_retries (default: 3) and database_retry_timeout (default: 5 seconds) settings.
6. V1.64.0 fixes runaway memory consumption when an extractor outputs records at a much higher rate than the loader can process them, by enabling flow control with a 64KB buffer size limit. 
7. V1.65.0 lets you tweak the size of the buffer that stores records output by the extractor (tap) while they are waiting to be processed by the loader (target), using a new elt.buffer_size setting.

(Earlier today, we also released V1.66.0!)

17 recent contributions by 11 community members

Done

1. Enable `pool_pre_ping` for the project’s DB engine by Suyash Behera (Goldman Sachs)
2. Update the ELT runner to explicitly log any error messages by Allan Whatmough (Run with AI)
3. Added sorting for `meltano select –list –all` by Nil
4. Added missing `mysql-logo.png` by Nil
5. Adding Poetry for better dependency and build management by Tobias Macey (MIT)
6. Make tap bigquery known to meltano by Niall Woodward (Tails.com)
7. Adding pre-commit and linting configurations by Tobias Macey (MIT)
8. Tap-salesforce:  Pass is_sandbox to authenticator by Kevin Ford
9. Files-dbt: Add bigquery profile by Daniel Pettersson (Smartr)
10. Linting fixes and contributor guide update by Tobias Macey (MIT)
11. Target-postgres: Fix non-null falsey values by Charles Julian Knight (FIXD)

In development

1. Search and replace “entities” >> “streams”, “attributes” >> “properties” by AJ Steers (Slalom)
2. Add pipx-based install instructions by AJ Steers (Slalom)
3. Singer SDK: Accelerated tap development framework (v0.0.1-alpha) by AJ Steers (Slalom)
4. Singer SDK: Initial mock-up for target-base by AJ Steers (Slalom)
5. Added libpq required target-postgres dependency install instructions by Geoff Langenderfer
6. Target-snowflake: Dynamic precision fix by Bryan Wise (Halosight)

Recent weekly Slack activity

As expected, activity dropped significantly over the holidays, but it’s steadily climbing back to our previous records of 157 “weekly active members” (Dec 9) and 30 “members who posted” (Nov 18).

Join us on Slack!

Other exciting recent and ongoing developments

• 18 community members created 41 new issues

• AJ Steers (Slalom) is working on the Singer SDK: a new framework and set of tools for building high-quality Singer taps and targets

• Tobias Macey (MIT, The Data Engineering Podcast) plans to build an importable Meltano library to let pipelines be run from arbitrary Python code
• Ken Payne (Tails.com) continues to work on a new BATCH message type to bring  PipelineWise’s Fast Sync feature to all taps and targets

• This quarter, we intend to hire 2 active contributors from the community onto the Meltano team at GitLab as Backend Engineers to work full-time on Meltano and related projects like the Singer SDK! If you’re interested, please reach out to project lead Douwe Maan on Slack.

This week’s priorities

Milestone issue board

• Features
  • Done: Make OUTPUT_BUFFER_SIZE configurable
• Bugs
  • Done: Jobs get stuck when `meltano elt` or UI worker running jobs is killed
• Documentation
  • Explain impact of `meltano {add,config}` commands on `meltano.yml` in Getting Started guide, so that user understands what’s happening behind the scenes
  • Move more concept documentation inline in Getting Started guide instead of only linking to other docs pages, which can feel overwhelming
  • Default transferwise variants of tap-postgres and tap-mysql are incompatible with default datamill-co variants of target-postgres and target-snowflake

Upcoming priorities

Milestones issue board

Epics:

• Documentation improvements
• Library of discoverable extractors and loaders
• Improved dbt integration
• ETL using Python scripts or YAML-defined rules
• Building and maintaining Singer taps and targets
• Debugging pipelines

Group Conversation Q&A