Building Meltano in Public: Six-weekly Recap

Last week, it was once again my turn to host a GitLab Group Conversation (a publicly live streamed Q&A on the GitLab Unfiltered YouTube channel) on Meltano!

I used the opportunity to share a recap of:

• the 5 releases we’ve had since the last update 6 weeks ago,
• 18 recent contributions made by 9 community members,
• recent weekly Slack activity,
• other exciting recent and ongoing developments,
• this week’s priorities, and
• a list of upcoming priorities.

If you’re curious, check out the presentation on Google Slides and the Q&A on YouTube. The presentation content is also reproduced below, as is an embedded video of the Q&A!

Group Conversation Presentation

5 releases since the last GC (2020-10-07)

1. V1.54.0 ensures arbitrary environment variables defined in .env are passed to invoked plugins, recreates plugin virtual environments when running meltano install to make sure the latest versions of dependencies are installed, and bumps the Airflow version to 1.10.12.
2. V1.55.0 adds out-of-the-box support for the datamill-co and transferwise (aka PipelineWise) variants of target-snowflake. The datamill-co variant is recommended for new users and is now the default, but the original meltano variant is still supported. 
3. V1.56.0 adds out-of-the-box support for the BigQuery destination using the target-bigquery loader, as well as official support for Python 3.8 alongside 3.6 and 3.7! 
4. V1.57.0 adds out-of-the-box support for the PostgreSQL and MySQL/MariaDB data sources using the transferwise (aka PipelineWise) variants of tap-postgres and tap-mysql.
5. V1.58.0 lets you disable autoscrolling in the pipeline log UI, fixes incorrectly disabled Test and Save buttons in the plugin configuration UI, and fixes logging in using Google Chrome when the UI is not served over HTTPS.

(Yesterday, we also released V1.59.0!)

18 recent contributions by 9 community members

Done

1. Rebuild virtual environments when running meltano install by Niall Woodward (Tails.com)
2. Correct docsDir configuration by Robert Speicher (GitLab)
3. Correct README `dev:docs` command to `serve:docs` by Robert Speicher (GitLab)
4. added OSerror catch for file system incompatibility by Nil
5. files-airflow: Added file exists check for meltano symlink, changed logging to logger by Nil
6. tap-gitlab:  Allow start_date to override state timestamp by Tom Elliff (CloudNC)
7. tap-gitlab:  docs: Default for GitLab.com is the v4 API by Zeger-Jan van de Weg (GitLab)
8. Resolve “Add `target-bigquery` loader” by Nil
9. Officially support py38 with ci testing by Niall Woodward (Tails.com)
10. Resolve “Add `singer-io` variant of `tap-recharge`” by Nil
11. Refactor discovery.yaml in lexicographically ascending order by Nil
12. Let user disable autoscrolling in UI job log modal by Niall Woodward (Tails.com)
13. Added check for date/datetime and convert to string for meltano config <plugin> by Niltap-facebook: Bump facebook business to `8.0.5` by Paul Blankley (Zenlytic)

In development

1. Enable `pool_pre_ping` for the project’s DB engine by Suyash Behera (Goldman Sachs)
2. Update the ELT runner to explicitly log any error messages by Allan Whatmough (Run with AI)
3. Search and replace “entities” >> “streams”, “attributes” >> “properties” by AJ Steers (Slalom)Add pipx-based install instructions by AJ Steers (Slalom)

Recent weekly Slack activity

In the graph on the below starting June 10, you can see how Slack activity has been growing steadily but dropped significantly during the week project lead Douwe Maan was out on vacation, with a low point on October 20. Fortunately, it recovered just as quickly once he returned!

Join us on Slack!

Other exciting recent and ongoing developments

• 16 community members created 20 new issues
• Srini Kadamati (Preset.io) wrote a guest blog post about building a Slack dashboard using Meltano & Superset
• Douwe Maan (Meltano project lead) hosted a talk and Q&A on open source EL(T) with Meltano and Singer: Recording, Slides

• Ken Payne (Tails.com) is building an equivalent of PipelineWise’s Fast Sync feature
• Eric Simmerman and interns (Qualytics) are building new and improved templates for Singer taps and targets
• Charles Julian Knight (FIXD) is looking into building a better loader for data lakes in S3 
• Tobias Macey (Data Engineering Podcast) is considering contributing support for the Dagster orchestrator
• We will be collaborating on built-in support for data validation with Great Expectations with the team at Superconductive
• AJ Steers (Slalom) is evaluating (and improving) the state of Windows (WSL) support

This week’s priorities

Milestone issue board

• Features
  • Allow plugins to inherit configuration from others
    • This replaces the Configuration Profiles feature considered earlier
• Documentation
  • Explain impact of `meltano {add,config}` commands on `meltano.yml` in Getting Started guide, so that user understands what’s happening behind the scenes
  • Move more concept documentation inline in Getting Started guide instead of only linking to other docs pages, which can feel overwhelming

Upcoming priorities

Milestones issue board

Epics:

• Improved dbt integration
• ETL using Python scripts or YAML-defined rules
• Building and maintaining Singer taps and targets

Group Conversation Q&A