Building Meltano in Public: Six-weekly Recap
Earlier this week, it was my turn to host a GitLab Group Conversation (a publicly live streamed Q&A on the GitLab Unfiltered YouTube channel) on Meltano.
I used the opportunity to share a recap of:
- the 7 releases we’ve had since the last update 5 weeks ago,
- 11 recent contributions made by 5 community members,
- recent weekly Slack activity,
- other exciting recent developments (including an upcoming talk on Oct 16!),
- current milestone priorities, and
- a list of epics for upcoming priorities.
If you’re curious, check out the presentation on Google Slides and the Q&A on YouTube. The presentation content is also reproduced below, as is an embedded video of the Q&A!
Group Conversation Presentation
Meltano has had 7 releases since the last GC (2020-09-01)
- V1.47.0 adds support for Bing Ads, prints docs and repo URLs when adding plugins, lets you specify a full schema for taps that can’t discover theirs, automatically uppercases target-snowflake‘s schema setting, and fixes a bug with embedded reports in the UI.
- V1.48.0 lets you extract a subset of selected entities using new
meltano elt, improves plugin invocation and extractor catalog discovery error messages, and changes where
meltano eltlogs and generated plugin config files are stored.
- V1.49.0 standardizes on
<PLUGIN_NAME>_<SETTING_NAME>for configuration environment variable names, makes environment variable expansion in setting values more flexible, and uses this to let you easily override your extractor’s
- V1.50.0 lets you manually provide extractor catalog and state files to
meltano eltusing new
stateextras), as an alternative to letting the catalog be generated on the fly and letting state be looked up based on the Job ID.
- V1.51.0 simplifies debugging extractor catalog generation, pipeline state lookup, and pipeline-specific configuration by letting you dump the contents of
meltano elt‘s generated catalog, state, and config files to STDOUT (or a file) using a new
- V1.52.0 fixes a bug where
meltano elt --transform=runwould unexpectedly install a transform plugin, and another where
meltano selectwould show outdated results after changing configuration.
- V1.53.0 lays the foundation for out-of-the-box support for different variants of extractors and loaders, like the
5 community members made 11 recent contributions
- Compose file update and readme addition by Nevin Morgan (VividFront)
- Override auth check when using a shared embed link by Allan Whatmough
- Resolve “Add a new `upcase_string` `value_processor` and apply to `target-snowflake`’s `schema` setting” by Nevin Morgan (VividFront)
- Add max_active_runs=1 to prevent scheduled job overlap by Niall Woodward (Tails.com)
- Remove automatic plugin install and remove associated tests by Paul Blankley (Zenlytic)
- Remove snowflake-connector-python dependency, bump snowflake-sqlalchemy, sqlalchemy and flask-sqlalchemy by Niall Woodward (Tails.com)
- Stop inheriting Meltano venv when invoking Airflow by Niall Woodward (Tails.com)
- Upgrade `pip` and related tools to the latest version in plugin venvs by Charles Julian Knight (FIXD)
- Add pipelinewise-tap-mysql and pipelinewise-target-snowflake to known tap and targets by Niall Woodward (Tails.com)
- Use pipenv for reproducible development environment by Niall Woodward (Tails.com)
- Bump Airflow version to 1.10.12 by Niall Woodward (Tails.com)
Recent weekly Slack activity
Other exciting recent developments
- 9 community members created 17 new issues
- Srini Kadamati wrote about building a Slack dashboard using Meltano & Superset
- Andrew Stewart, “Intro to data science” lecturer at Johns Hopkins University, wrote about “Data Stacks For Fun & Nonprofit” with Meltano, dbt & Superset
- Andrew is considering adding Meltano to the curriculum for the spring semester!
- Project lead Douwe Maan was invited to talk about “Open source EL(T) with Meltano and Singer” at a virtual event hosted by the Data Nerd Herd on October 16
- More details and free registration: Register for Zoom Meeting
- The TransferWise data team will be collaborating with the Meltano community on bringing PipelineWise’s DB-to-DB Fast Sync feature to Meltano, so that PipelineWise can eventually be phased out in favor of Meltano, at TransferWise and elsewhere
Current milestone priorities
- Epic: Expand library of discoverable extractors and loaders supported out of the box
Epics for upcoming priorities
Group Conversation Q&A
Category: Month in Review