What happened in the Meltano universe in the last four weeks? Tap-Toberfest, multiple new releases and many contributions! Let’s dive into the details!
Tap-Toberfest happened! And it was a blast.
Thanks to all the great contributors, participants, people just checking us out during this period. Thanks for contributions, feedback and good vibes. We highlight a lot of the contributions below! We had fun, and will continue to celebrate the Tap-Toberfest next year.
What got shipped?
We shipped two major releases that added great features like initializing into empty and directories and making the .meltano path configurable. Both of these features make it easier to get started with Meltano.
We also added an option to override the working directory to make plugins even more powerful and simpler to use.
List of all releases
- v2.8.0 You can now run “meltano init .” into the current (empty) directory. And read the improved custom extractor creation tutorial inside the docs. Includes 3 community contributions (thanks to Reuben Frankel, Martin Morset, and Jared Rimmer!).
- v2.9.0 You can override the working directory, which is useful when working with some plugins, configure the location of the “.meltano” folder and turn off colored output using the env. variable NO_COLOR=True. Includes 4 community contributors (thanks to rawwar, BuzzCutNorman, kayakr and jx2lee!)
Community & Hub updates
- Jonas Kalderstam created a tap-bigquery with BATCH support!
- utility-dagster hit the hub, with cloud support (Jules Huisman thanks!).
- Utility-gdrive appeared on the hub as well (Daniel Walker).
- Utility-postgres, a local postgres, got added to the hub (Aaron Phethean).
- And the utility-notebook finally got its place on the hub (Reuben Frankel).
- Not to mention that we got almost 100 plugin settings & metadata updated on the hub!
The Community in Action
This month the community provided everything including core Meltano features, plugins, blog articles and GitHub repositories implementing Meltano in action. Here are a few selected highlights:
- Jacob Matson wrote an impressive article on building a modern data stack “in a box” using meltano & duckDB. Also be sure to check out the accompanying GitHub repository.
- Jonas Kalderstam didn’t just create the tap-bigquery with BATCH support, he also wrote an article showing the performance improvements: 6x!
- We cut out a short clip of Andrew Stewart showing off how to use the machine learning library Kedro together with Meltano.
Join the Conversation
The Meltano community is active on a variety of GitHub repositories like the core meltano/meltano repository as well as on Slack. Here are a few conversations that might interest you!
- Want to use the Meltano map transformer to rename columns? This thread provides more guidance.
- People are also integrating Python scripts into their Meltano workflows. This thread discusses several different options on doing this.
- Your source data keeps on changing, do you query the just the new data or do you delete the old one?
- Looking for EL best practices? Or do you have any to share? Then participate in this thread & brainstorm!
Additional Updates
A lot has happened on the Meltano blog, here are a few highlights.
- We published an article on the key differences between PostgreSQL and MySQL.
Keep on being Meltastic! And if we missed your contribution to the community, just hit us up on Slack, LinkedIn or wherever.