Building Meltano in Public: 6-weekly recap

Earlier this week, it was my turn to host a GitLab Group Conversation (a publicly live streamed Q&A on the GitLab Unfiltered YouTube channel) on Meltano. Just like last time, I used the opportunity to provide a recap of what’s happened in the community and the progress that’s been made since the last update 6 weeks ago.

If you’re curious, check out the presentation on Google Slides and the Q&A on YouTube. The presentation content is also reproduced below, as is an embedded video of the Q&A!

Group Conversation Presentation

12 releases since the last GC (2020-06-09), with 72 changelog items!

1. V1.35.0 (10 changelog items) makes it easier to manage and version control a Meltano project’s configuration by having meltano config store values in meltano.yml by default, instead of the system database.
2. V1.35.1 (3 changelog items) fixes 2 minor bugs and makes the output of meltano add and meltano install consistent in how they refer to plugin types and names.
3. V1.36.0 (7 changelog items) introduces “file bundle” plugins that are used to automatically add plugin-specific files to your Meltano project when adding a plugin like the dbt transformer or Airflow orchestrator.
4. V1.36.1 (5 changelog items) adds the ability to specify Singer tap stream and property metadata as if it were any other plugin configuration.
5. V1.37.0 (4 changelog items) lets you instantly containerize your Meltano project using meltano add files docker.
6. V1.38.0 (5 changelog items) helps you debug your plugin configuration by having meltano config <plugin> list show the current values of settings and where they come from, alongside the setting names and their environment variables.
7. V1.38.1 (8 changelog items) contains a significant 61-file refactor of how project settings and environment variables are managed internally.
8. V1.39.0 (6 changelog items) enables more streamlined and consistent configuration of all Meltano and Meltano UI settings using meltano config and the same highly flexible layered configuration model already used to manage plugin settings.
9. V1.39.1 (9 changelog items) adds various new settings that make it easier to customize Meltano UI and configure it for production.
10. V1.40.0 (1 changelog item) lets you customize how your custom extractors show up in Meltano UI using a new logo_url property and existing label and description properties. 
11. V1.40.1 (9 changelog items) makes meltano config and the UI automatically store configuration in the most appropriate location, taking into account whether the project is deployed as read-only, and whether the setting is sensitive or environment-specific. 
12. V1.41.0 (6 changelog items) adds a --full-refresh flag to meltano elt to perform a full refresh, ignoring any state left behind by previous runs with the same job ID.

Slack activity over the last 30 days

Join us on Slack!

Slack activity since May 1

I’m not sure why we’re seeing a downward trend in weekly active members over the last week, but if that hasn’t turned around by the next GC, I promise I’ll have more to say about it then!

Other things that have happened since the last GC (2020-06-09)

• 11 community members filed 23 new issues (19 in the last 2 weeks)
• 5 community members contributed 8 MRs
  • RFA contributed 4 MRs to address issues in the Data pipeline management UI epic
• A community member is currently working on a Helm chart to deploy Meltano to Kubernetes
• The Data Engineering Podcast episode Open Source Production Grade Data Integration With Meltano went live
  • Feedback from a community member: I tried Meltano on my Windows with WSL2 and apart from few glitches the experience was really good. So far I only tried the example on the site but looking forward to start playing around with other taps and targets. I tried Meltano couple of months back and the experience was not that great so I left, but after I heard @Douwe Maan on Data Engineering Podcast and the changed vision, I gave it a shot again. I’m really excited to learn, apply and contribute.
• Meltano was covered in Airflow Summit talk Building reusable and trustworthy ELT pipelines (A templated approach)

Current milestone

This week, I’ll be working on these issues filed or requested by the community:

• Features
  • Add option to modify schema properties of a stream in meltano.yml
• Bugs
  • Interrupting a pipeline leaves it in the running state
  • Persist and consider state messages regardless of job success
  • ELT run fails when more than 64KiB of state is written
• Documentation
  • Document how to use different settings for different pipelines of the same tap/target
  • Restructure and update CLI documentation
  • Document `meltano.yml` format

RFA is expected to work on:

• List installed and available extractors separately on Extractors page
• Add new Loaders page to UI

Group Conversation Q&A