Building Meltano in Public: 6-weekly recap

Earlier this week, it was my turn to host a GitLab Group Conversation (a publicly live streamed Q&A on the GitLab Unfiltered YouTube channel) on Meltano.

Just like last time, I used the opportunity to provide a recap of the progress that’s been made since the last update 6 weeks ago, and to share our priorities for the upcoming weeks.

If you’re curious, check out the presentation on Google Slides and the Q&A on YouTube. The presentation content is also reproduced below, as is an embedded video of the Q&A!

Group Conversation Presentation

6 releases since the last GC (2020-07-21)

  1. V1.41.1 lets you easily override Singer stream schema descriptions for specific attributes (e.g. {“type”: [“string”, “null”], “format”: “date-time”}) using a new schema extractor extra.
  2. V1.42.0 makes meltano elt and its state management more resilient to failure and interruption.
  3. V1.43.0 improves the output of meltano elt in three significant ways.
  4. V1.44.0 fixes a bug that caused meltano elt to fail when an extractor (Singer tap) emits a RECORD message larger than 64KB in size.
  5. V1.45.0 adds a new Loaders tab to Meltano UI that lets you easily manage and configure your project’s loaders, as an alternative to using the meltano discover, meltano add, and meltano config CLI commands.
  6. V1.46.0 brings out-of-the-box support for extractor tap-spreadsheets-anywhere (docs), which lets you pull data from CSV and Excel files stored locally or in the cloud (S3, SFTP, HTTP, etc).

A lot of work was also done on reorganizing the documentation, which brought us: 

16 community contributions since the last GC (2020-07-21)

Done

  1. List installed and available extractors separately on Extractors page by Dmitry Stadnik (RFA)
  2. Update Docker file for faster builds by Paul Blankley
  3. Add Jobs resource to tap-gitlab by Charles Julian Knight
  4. Add Loaders page to UI by Dmitry Stadnik (RFA)
  5. Properly handle columns with reserved names in target-snowflake by Eric Simmerman
  6. Add oauth_credentials to tap-google-analytics settings in discovery.yml by Zachary Wynegar
  7. Use `singer-io/target-csv` instead of our own fork by Zachary Wynegar
  8. Add tap-spreadsheets-anywhere by Eric Simmerman

In review

  1. Create a JSON schema for `meltano.yml` and publish it on schemastore.org by Zachary Wynegar
  2. Improve docker-compose file template by Nevin Morgan

In development

  1. Allow system database schema to be overridden when PostgreSQL is used by Charles Julian Knight
  2. Upgrade `facebook_business` to 8.0.0 in tap-facebook by Zachary Wynegar
  3. Add support for `<ENV>_FILE` config env vars to support Docker Secrets by Zachary Wynegar
  4. Add Google Cloud Composer tutorial by Paul Tiplady
  5. Override auth check when using a shared embed link by Allan Whatmough
  6. Allow extractor entities to be selected in UI by Dmitry Stadnik (RFA)

Upcoming milestone priorities

Milestone issue board

Features

Bugs

Documentation

Epics for upcoming priorities

(Exact order to be determined)

Slack activity over the last 30 days

Join us on Slack!

Group Conversation Q&A