Building Meltano in Public: 6-weekly recap

Last week, it was once again my turn to host a GitLab Group Conversation (a publicly live streamed Q&A on the GitLab Unfiltered YouTube channel) on Meltano!

I used the opportunity to share a recap of:

If you’re curious, check out the presentation on Google Slides and the Q&A on YouTube. The presentation content is also reproduced below, as is an embedded video of the Q&A!

Group Conversation Presentation

5 releases since the last GC (2020-10-07)

  1. V1.54.0 ensures arbitrary environment variables defined in .env are passed to invoked plugins, recreates plugin virtual environments when running meltano install to make sure the latest versions of dependencies are installed, and bumps the Airflow version to 1.10.12.
  2. V1.55.0 adds out-of-the-box support for the datamill-co and transferwise (aka PipelineWise) variants of target-snowflake. The datamill-co variant is recommended for new users and is now the default, but the original meltano variant is still supported. 
  3. V1.56.0 adds out-of-the-box support for the BigQuery destination using the target-bigquery loader, as well as official support for Python 3.8 alongside 3.6 and 3.7! 
  4. V1.57.0 adds out-of-the-box support for the PostgreSQL and MySQL/MariaDB data sources using the transferwise (aka PipelineWise) variants of tap-postgres and tap-mysql.
  5. V1.58.0 lets you disable autoscrolling in the pipeline log UI, fixes incorrectly disabled Test and Save buttons in the plugin configuration UI, and fixes logging in using Google Chrome when the UI is not served over HTTPS.

(Yesterday, we also released V1.59.0!)

18 recent contributions by 9 community members

Done

  1. Rebuild virtual environments when running meltano install by Niall Woodward (Tails.com)
  2. Correct docsDir configuration by Robert Speicher (GitLab)
  3. Correct README `dev:docs` command to `serve:docs` by Robert Speicher (GitLab)
  4. added OSerror catch for file system incompatibility by Nil
  5. files-airflow: Added file exists check for meltano symlink, changed logging to logger by Nil
  6. tap-gitlab:  Allow start_date to override state timestamp by Tom Elliff (CloudNC)
  7. tap-gitlab:  docs: Default for GitLab.com is the v4 API by Zeger-Jan van de Weg (GitLab)
  8. Resolve “Add `target-bigquery` loader” by Nil
  9. Officially support py38 with ci testing by Niall Woodward (Tails.com)
  10. Resolve “Add `singer-io` variant of `tap-recharge`” by Nil
  11. Refactor discovery.yaml in lexicographically ascending order by Nil
  12. Let user disable autoscrolling in UI job log modal by Niall Woodward (Tails.com)
  13. Added check for date/datetime and convert to string for meltano config <plugin> by Niltap-facebook: Bump facebook business to `8.0.5` by Paul Blankley (Zenlytic)

In development

  1. Enable `pool_pre_ping` for the project’s DB engine by Suyash Behera (Goldman Sachs)
  2. Update the ELT runner to explicitly log any error messages by Allan Whatmough (Run with AI)
  3. Search and replace “entities” >> “streams”, “attributes” >> “properties” by AJ Steers (Slalom)Add pipx-based install instructions by AJ Steers (Slalom)

Recent weekly Slack activity

In the graph on the below starting June 10, you can see how Slack activity has been growing steadily but dropped significantly during the week project lead Douwe Maan was out on vacation, with a low point on October 20. Fortunately, it recovered just as quickly once he returned!

Join us on Slack!

Other exciting recent and ongoing developments

This week’s priorities

Milestone issue board

Upcoming priorities

Milestones issue board

Epics:

Group Conversation Q&A