Building Meltano in Public: Six-weekly Recap

Earlier this week, it was my turn to host a GitLab Group Conversation (a publicly live streamed Q&A on the GitLab Unfiltered YouTube channel) on Meltano.

I used the opportunity to share a recap of:

• the 7 releases we’ve had since the last update 5 weeks ago,
• 11 recent contributions made by 5 community members,
• recent weekly Slack activity,
• other exciting recent developments (including an upcoming talk on Oct 16!),
• current milestone priorities, and
• a list of epics for upcoming priorities.

If you’re curious, check out the presentation on Google Slides and the Q&A on YouTube. The presentation content is also reproduced below, as is an embedded video of the Q&A!

Group Conversation Presentation

Meltano has had 7 releases since the last GC (2020-09-01)

1. V1.47.0 adds support for Bing Ads, prints docs and repo URLs when adding plugins, lets you specify a full schema for taps that can’t discover theirs, automatically uppercases target-snowflake‘s schema setting, and fixes a bug with embedded reports in the UI.
2. V1.48.0 lets you extract a subset of selected entities using new --select and --exclude options on meltano elt, improves plugin invocation and extractor catalog discovery error messages, and changes where meltano elt logs and generated plugin config files are stored. 
3. V1.49.0 standardizes on <PLUGIN_NAME>_<SETTING_NAME> for configuration environment variable names, makes environment variable expansion in setting values more flexible, and uses this to let you easily override your extractor’s load_schema.
4. V1.50.0 lets you manually provide extractor catalog and state files to meltano elt using new --catalog and --state options (and catalog and state extras), as an alternative to letting the catalog be generated on the fly and letting state be looked up based on the Job ID.
5. V1.51.0 simplifies debugging extractor catalog generation, pipeline state lookup, and pipeline-specific configuration by letting you dump the contents of meltano elt‘s generated catalog, state, and config files to STDOUT (or a file) using a new --dump option.
6. V1.52.0 fixes a bug where meltano elt --transform=run would unexpectedly install a transform plugin, and another where meltano select would show outdated results after changing configuration.
7. V1.53.0 lays the foundation for out-of-the-box support for different variants of extractors and loaders, like the transferwise and datamill-co variants of target-snowflake and the singer-io variant of tap-facebook.  

5 community members made 11 recent contributions

Done

1. Compose file update and readme addition by Nevin Morgan (VividFront)
2. Override auth check when using a shared embed link by Allan Whatmough
3. Resolve “Add a new `upcase_string` `value_processor` and apply to `target-snowflake`’s `schema` setting” by Nevin Morgan (VividFront)
4. Add max_active_runs=1 to prevent scheduled job overlap by Niall Woodward (Tails.com)
5. Remove automatic plugin install and remove associated tests by Paul Blankley (Zenlytic)
6. Remove snowflake-connector-python dependency, bump snowflake-sqlalchemy, sqlalchemy and flask-sqlalchemy by Niall Woodward (Tails.com)
7. Stop inheriting Meltano venv when invoking Airflow by Niall Woodward (Tails.com)
8. Upgrade `pip` and related tools to the latest version in plugin venvs by Charles Julian Knight (FIXD)

In development

1. Add pipelinewise-tap-mysql and pipelinewise-target-snowflake to known tap and targets by Niall Woodward (Tails.com)
2. Use pipenv for reproducible development environment by Niall Woodward (Tails.com)
3. Bump Airflow version to 1.10.12 by Niall Woodward (Tails.com)

Recent weekly Slack activity

Join us on Slack!

Other exciting recent developments

• 9 community members created 17 new issues
• Srini Kadamati wrote about building a Slack dashboard using Meltano & Superset
  • https://preset.io/blog/2020-09-22-slack-dashboard/
  • https://preset.io/blog/2020-10-02-slack-dashboard-part-2/
• Andrew Stewart, “Intro to data science” lecturer at Johns Hopkins University, wrote about “Data Stacks For Fun & Nonprofit” with Meltano, dbt & Superset
  • https://medium.com/@andrewcstewart/data-stacks-for-fun-nonprofit-part-ii-d375d824abf3
  • Andrew is considering adding Meltano to the curriculum for the spring semester!
• Project lead Douwe Maan was invited to talk about “Open source EL(T) with Meltano and Singer” at a virtual event hosted by the Data Nerd Herd on October 16
  • More details and free registration: Register for Zoom Meeting
• The TransferWise data team will be collaborating with the Meltano community on bringing PipelineWise’s DB-to-DB Fast Sync feature to Meltano, so that PipelineWise can eventually be phased out in favor of Meltano, at TransferWise and elsewhere
  • https://gitlab.com/meltano/meltano/-/issues/2364

Current milestone priorities

Milestone issue board

• Epic: Expand library of discoverable extractors and loaders supported out of the box
  • Add `transferwise` and `datamill-co` variants of `target-snowflake`
  • Add `singer-io` variant of tap-facebook
  • Add `target-bigquery` loader
  • Add support for all PipelineWise taps and targets
• Bugs
  • Upgrade tap-facebook `facebook_business` to 8.0.0 since Marketing API v5 is no longer available
  • `meltano install` doesn’t install new version of plugin when `pip_url` is changed but package version stayed the same
• Documentation
  • Format extractor and loader docs consistently
  • Add “Using a custom fork of a plugin” section to Plugin Management doc
  • Add “Removing plugins from your project” section to Plugin Management doc

Epics for upcoming priorities

• Improved dbt integration
• ETL using Python scripts or YAML-defined rules
• Plugin Configuration Profiles

Group Conversation Q&A