Building Meltano in Public: 6-weekly recap

Earlier this week, it was my turn to host a GitLab Group Conversation (a publicly live streamed Q&A on the GitLab Unfiltered YouTube channel) on Meltano.

I used the opportunity to share a recap of:

If you’re curious, check out the presentation on Google Slides and the Q&A on YouTube. The presentation content is also reproduced below, as is an embedded video of the Q&A!

Group Conversation Presentation

Meltano has had 7 releases since the last GC (2020-09-01)

  1. V1.47.0 adds support for Bing Ads, prints docs and repo URLs when adding plugins, lets you specify a full schema for taps that can’t discover theirs, automatically uppercases target-snowflake‘s schema setting, and fixes a bug with embedded reports in the UI.
  2. V1.48.0 lets you extract a subset of selected entities using new --select and --exclude options on meltano elt, improves plugin invocation and extractor catalog discovery error messages, and changes where meltano elt logs and generated plugin config files are stored. 
  3. V1.49.0 standardizes on <PLUGIN_NAME>_<SETTING_NAME> for configuration environment variable names, makes environment variable expansion in setting values more flexible, and uses this to let you easily override your extractor’s load_schema.
  4. V1.50.0 lets you manually provide extractor catalog and state files to meltano elt using new --catalog and --state options (and catalog and state extras), as an alternative to letting the catalog be generated on the fly and letting state be looked up based on the Job ID.
  5. V1.51.0 simplifies debugging extractor catalog generation, pipeline state lookup, and pipeline-specific configuration by letting you dump the contents of meltano elt‘s generated catalog, state, and config files to STDOUT (or a file) using a new --dump option.
  6. V1.52.0 fixes a bug where meltano elt --transform=run would unexpectedly install a transform plugin, and another where meltano select would show outdated results after changing configuration.
  7. V1.53.0 lays the foundation for out-of-the-box support for different variants of extractors and loaders, like the transferwise and datamill-co variants of target-snowflake and the singer-io variant of tap-facebook.  

5 community members made 11 recent contributions

Done

  1. Compose file update and readme addition by Nevin Morgan (VividFront)
  2. Override auth check when using a shared embed link by Allan Whatmough
  3. Resolve “Add a new `upcase_string` `value_processor` and apply to `target-snowflake`’s `schema` setting” by Nevin Morgan (VividFront)
  4. Add max_active_runs=1 to prevent scheduled job overlap by Niall Woodward (Tails.com)
  5. Remove automatic plugin install and remove associated tests by Paul Blankley (Zenlytic)
  6. Remove snowflake-connector-python dependency, bump snowflake-sqlalchemy, sqlalchemy and flask-sqlalchemy by Niall Woodward (Tails.com)
  7. Stop inheriting Meltano venv when invoking Airflow by Niall Woodward (Tails.com)
  8. Upgrade `pip` and related tools to the latest version in plugin venvs by Charles Julian Knight (FIXD)

In development

  1. Add pipelinewise-tap-mysql and pipelinewise-target-snowflake to known tap and targets by Niall Woodward (Tails.com)
  2. Use pipenv for reproducible development environment by Niall Woodward (Tails.com)
  3. Bump Airflow version to 1.10.12 by Niall Woodward (Tails.com)

Recent weekly Slack activity

Join us on Slack!

Other exciting recent developments

Current milestone priorities

Milestone issue board

Epics for upcoming priorities

Group Conversation Q&A