Building Meltano in Public: 6-weekly recap
Earlier this week, it was my turn to host a GitLab Group Conversation (a publicly live streamed Q&A on the GitLab Unfiltered YouTube channel) on Meltano. Just like last time, I used the opportunity to provide a recap of what’s happened in the community and the progress that’s been made since the last update 6 weeks ago.
If you’re curious, check out the presentation on Google Slides and the Q&A on YouTube. The presentation content is also reproduced below, as is an embedded video of the Q&A!
Group Conversation Presentation
12 releases since the last GC (2020-06-09), with 72 changelog items!
- V1.35.0 (10 changelog items) makes it easier to manage and version control a Meltano project’s configuration by having
meltano configstore values in
meltano.ymlby default, instead of the system database.
- V1.35.1 (3 changelog items) fixes 2 minor bugs and makes the output of
meltano installconsistent in how they refer to plugin types and names.
- V1.36.0 (7 changelog items) introduces “file bundle” plugins that are used to automatically add plugin-specific files to your Meltano project when adding a plugin like the dbt transformer or Airflow orchestrator.
- V1.36.1 (5 changelog items) adds the ability to specify Singer tap stream and property metadata as if it were any other plugin configuration.
- V1.37.0 (4 changelog items) lets you instantly containerize your Meltano project using
meltano add files docker.
- V1.38.0 (5 changelog items) helps you debug your plugin configuration by having
meltano config <plugin> listshow the current values of settings and where they come from, alongside the setting names and their environment variables.
- V1.38.1 (8 changelog items) contains a significant 61-file refactor of how project settings and environment variables are managed internally.
- V1.39.0 (6 changelog items) enables more streamlined and consistent configuration of all Meltano and Meltano UI settings using
meltano configand the same highly flexible layered configuration model already used to manage plugin settings.
- V1.39.1 (9 changelog items) adds various new settings that make it easier to customize Meltano UI and configure it for production.
- V1.40.0 (1 changelog item) lets you customize how your custom extractors show up in Meltano UI using a new
logo_urlproperty and existing
- V1.40.1 (9 changelog items) makes
meltano configand the UI automatically store configuration in the most appropriate location, taking into account whether the project is deployed as read-only, and whether the setting is sensitive or environment-specific.
- V1.41.0 (6 changelog items) adds a
meltano eltto perform a full refresh, ignoring any state left behind by previous runs with the same job ID.
Slack activity over the last 30 days
Slack activity since May 1
I’m not sure why we’re seeing a downward trend in weekly active members over the last week, but if that hasn’t turned around by the next GC, I promise I’ll have more to say about it then!
Other things that have happened since the last GC (2020-06-09)
- 11 community members filed 23 new issues (19 in the last 2 weeks)
- 5 community members contributed 8 MRs
- RFA contributed 4 MRs to address issues in the Data pipeline management UI epic
- A community member is currently working on a Helm chart to deploy Meltano to Kubernetes
- The Data Engineering Podcast episode Open Source Production Grade Data Integration With Meltano went live
- Feedback from a community member: I tried Meltano on my Windows with WSL2 and apart from few glitches the experience was really good. So far I only tried the example on the site but looking forward to start playing around with other taps and targets. I tried Meltano couple of months back and the experience was not that great so I left, but after I heard @Douwe Maan on Data Engineering Podcast and the changed vision, I gave it a shot again. I’m really excited to learn, apply and contribute.
- Meltano was covered in Airflow Summit talk Building reusable and trustworthy ELT pipelines (A templated approach)
This week, I’ll be working on these issues filed or requested by the community:
RFA is expected to work on: