Meltastic by Meltano: February 2023 Edition
What happened in the Meltano universe in the last four weeks? Meltano Cloud is soon going into Private Beta, so be sure to get on the waiting list! Oh and we shipped 7 releases, lots of connectors got added to the hub and much more.
—
Let’s get to the details.
Table of Contents
⭐Meltano cloud: you build pipelines, we keep them running!⭐
TL;DR: Sign up for the waitlist now to be among the first to try out Meltano Cloud!
The biggest thing we’ve been working on over the past months has been Meltano Cloud: the easiest way to get your Meltano project into production, with all the flexibility, control and transparency we’re known for, but none of the hassle of setting up, maintaining, and scaling your own infrastructure.
Plus: extremely predictable pricing based on the number of pipelines you run in a month independent of changeable and growing data volumes, professional support, and new connectors for common (high-volume) data sources built and maintained by our team and consulting partners. If you were ever going to move your remaining pipelines off of Fivetran, Stitch or other SaaS EL tools to save cost, increase productivity, and gain flexibility… this is the moment you’ve been waiting for 🙂
Meltano Cloud is currently in Private Alpha with a select number of customers and consultancies, and we’re excited to be opening the doors to more of you when we go into Private Beta on March 28, followed by Public Beta in April and General Availability in May. If you’d like to be the first to try it out (or simply learn more), sign up for the waitlist and we’ll set up some time to dive deeper into your EL needs and how Meltano Cloud could fit in!
What got shipped?
We shipped seven releases in the past 30 days. Four for the SDK, and three for Meltano. The sdk now has in-built testing for taps & targets making connectors more robust and easier to develop. Meltano itself now supports Python 3.11, has better documentation and finally a way to compile your complete project into one file.
For everyone looking for single instance or single long-running container deployments of Meltano, there’s now a neat cron scheduler you can use to make these really simple.
List of all releases:
- Meltano v.2.16.0 & v.2.16.1 provide you with the ability to “compile” your project into one big manifest, including all your different project files into one.
- Meltano v.2.15.0–v.2.5.3 adds support for Python 3.11!
- Meltano v.2.14.0 improves the documentation and makes the external state backends run smoother.
- SDK v.0.21.0 fixes a few tap templating issues.
- Sdk v.0.20.0 adds a bunch of utilities.
- Sdk v.0.19.0 improves the included testing for taps & targets, this makes connector development way easier and more robust.
- Sdk v.0.18.0 adds automatic catalog selection of replication keys. The replication key fields (e.g. created_at) are now extracted even if the user doesn’t select them explicitly. This makes taps more beginner friendly and less failure prone.
- All of this got shipped in part thanks to the community contributors: @stkbailey, @flexponsive, @BuzzCutNorman,@qbatten, @menzenski,@chodera & @jx2lee!
Community & Hub updates:
- Connor Flynn just added a tap-ga-4 to the Hub, thanks!
- Alex Butler created a tap-mongodb based on the SDK, nice work!
- He also added target-bigquery with all the fancy batch support etc. Way to go.
- Thomas Briggs created a target-jinja, crazy stuff.
- Tap-cloudwatch got added to the hub.
- There now is a target-jsonl and tap-jsonl connector on the hub.
- We now have a cron scheduler on the hub, that’s a pretty cool thing as you now have a sweet alternative to airflow or “hand cron” for EC2 instances or long running single container deployments of Meltano.
- And we have another target-s3-parquet variant on the hub now.
- Stèphane Burwash added a dbt-dry-run utility to the hub, cool!
- We got a bunch of newcomers & variants to the hub: target-snowflake (meltanolabs), target-s3 (crowemi), tap-emarsys (gthesheep), tap-segment (gthesheep), tap-gong (symon-ai), tap-qualified (z3z1ma) and tap-showpad (z3z1ma)….
Meltano Related Content
A lot is happening around Meltano on the web. We’re selecting a few pieces here. Be sure to check them out if they sound interesting.
- Our data engineer Pat is doing a fantastic job making our ingestion pipelines unbreakable.
- Pat also did a video demo on using environments in a complete dev workflow a while back, still worth gold!
- We shared all the different ways of handling PII inside data ingestion and how to implement them using Meltano.
- Jacek from GoodData held a Meet Up on using Meltano together with GoodData and CI/CD pipelines.
- Jacek also wrote up a great article outlining his talk, go check it out!
- Matt from PayIt shared how to build dbt docs as part of a meltano pipeline running inside GitHub Actions.
- We shared our favorite reading recommendations for 2023.
Keep on being Meltastic! And if we missed your contribution to the community, just hit us up on Slack, LinkedIn or wherever.
