What happened in the Meltano universe in the last four weeks? The Meltano blog was filled with a couple of great articles, including the community’s favorite dbt packages and the community’s practices for extract and load.
We added a codespaces-powered Meltano demo, and much more!
Now, let’s jump into the details!
⭐Extracting trash from the ocean⭐
Most importantly, we’re extracting trash & transforming our waterways in December!
=> Just star ⭐⭐⭐ our Meltano repository to remove 2 lbs of trash for every star from the ocean.
What got shipped?
We shipped two major releases and one minor one. Great features got added to Meltano, one of the biggest being support for remote state backend on S3, GCS, Azure or your local file system.
List of all releases:
- v2.11.0 & v2.11.1 Add a `–force` flag to `meltano install` to ignore python version constraints from plugins. You can also now include environment variables inside pip urls for plugins to e.g. encode credentials or versions.
- Includes 2 community contributions (thanks to Quinn Batten and EMCP!)
- v2.10.0 We’re very excited about this release, it enables you to host your state backend on all major object stores like AWS S3 or GCS. This means you can easily deploy Meltano to any managed orchestrator like prefect, dagster, airflow and have it work without having a large postgres instance ready.
- Includes 1 community contribution (thanks to Daniel Walker!).
Community & Hub updates:
- There’s a new utility on the hub for elementary adding observability to dbt (thanks Stéphane Burwash).
- A tap for stackoverflow sample data also made it to the hub! (Thank you Dan Norman)
- Niall Woodward set up a GitHub Actions based meltano project, thank you Niall.
- We added Clickhouse as a utility to the hub.
- We also added 15 new extractors and a bunch of variants to the hub.
- Steve Clarke added a new tap-oracle to the hub, thanks!
Join the Conversation 💬
The Meltano community is active on a variety of GitHub repositories like the core meltano/meltano repository as well as on Slack. Here are a few conversations that might interest you!
- Replicating data from snowflake to S3? Here’s a discussion on copying only partial tables.
- What are your metrics to measure data engineering teams? Here’s an ongoing discussion.
- In a Slack brainstorm we already came up with 12 different kinds of types and names for data warehouse tables! Add yours, or read through the ones we already collected.
VS Code & Meltano
We also added a short explanation of using VS Code and Meltano together featuring two cool extensions that will make your life much easier.
Additional Updates 📰
A lot has happened on the Meltano blog, here are a few highlights.
- 5 Helpful Extract & Load Practices for High-Quality Raw Data
- The Meltano Communities’ Surprising Top 5 dbt Packages and Tools in 2022
Keep on being Meltastic! And if we missed your contribution to the community, just hit us up on Slack, LinkedIn or wherever.