What happened in the Meltano universe in the last four weeks? A ton!
Using Meltano To Power Products
Traditionally, most people use Meltano in their internal data platform to integrate data for BI use cases. But internal analytics dashboards aren’t the only thing you can build on third-party data: a growing number in our community are instead building products, product features, and ML training pipelines powered by data from their customers’ DBs and third-party APIs rather than their own, with Meltano functioning as an embedded data integration engine that is managed programmatically, rather than a stand-alone tool that users interact with directly. Meltano is already an excellent fit for this use case because of its declarative code-first approach, open source nature, inherent programmability, and infinite customizability enabled by the Meltano SDK and features like inline stream maps. Still, as this hasn’t been our primary use case, there are some rough edges as well, and it’s clear it wasn’t optimized to be used that way. We’d love to change that and make Meltano (Cloud) the best possible solution for all product & software teams that are building data-powered & ML products & features and need efficient programmatic access to all the third-party data that powers them, whether those are dozen internal sources or one API multiplied by 1000 customer credentials. We have several exciting features for this use case on the roadmap already, and the pricing model for Meltano Cloud is also being redesigned to be more predictable (and less expensive) on a per-customer basis, considering the larger number of pipelines and more frequent syncs than we originally had in mind. We’d love to hear from you if you’re using Meltano to do something other than internal data analytics and manage it programmatically rather than manually. Feel free to respond to this email, send a Slack DM to our CEO @Douwe Maan, or book some time to chat directly on his calendar.
Wait, Meltano can do that?
Meltano is only suitable for internal data analytics pushing data into data warehouses? Not so fast.
Meltano can be used to build any pipeline quickly.
Take the Tidbyt, for example, a pixel screen that displays apps that are first built on a server (in webp form) and then pushed onto the device. This is a pipeline-like process, right? That’s exactly what our CEO Douwe built as a side-project: tap-pixlet, target-tidbyt, and a Meltano project that ties them together named Pixbyt: “a self-hosted Tidbyt app server for advanced apps.” Tidbyt users can fork the project, add their apps, and spin it up on their homelab without knowing that it’s Meltano and Singer doing the heavy lifting!
Did you find any novel uses for Meltano? We’d love to hear from you – just hit us up on Slack or any of our other social media.
>>> Check out the full article here! <<<
What got shipped?
We’re seeing many more high-quality connectors on the hub (e.g., the Hubspot ones!), and now have a great intro video for configuring target-snowflake, the most used connector.
Thanks to our contributor stars this month: @BuzzCutNorman, @mjsqu, @s7clarke10, @riordan
Meltano Cloud
The Cloud CLI is progressing fast! Check out the docs for more details:
- Added the ability to cancel a running job from the CLI
- Added syntax highlighting to the logs in the Cloud UI
- Added support for manual, once, and none aliases for schedules. Useful for one-off runs!
- Improved the error code response when creating or updating a deployment
- Squashed quite a few bugs. Thanks to our early users for all of their feedback!
SDK, Meltano
The Meltano release v2.20.0 features bring dozens of docs improvements and a new hidden flag (while retaining the type information).
The Meltano SDK releases v0.30.0 and v0.29.0 feature many target improvements, including better batch messaging handling and more extensive tests. Union schemas are now supported. This is, e.g., useful for taps that allow for multiple authentication methods.
Community & Hub updates
This month we again saw much action on the hub.
- We now have an intro video for the target-snowflake! Setting up the snowflake target was never easier.
- A HTTP endpoint target.
- A tap for Cassandra.
- A new variation of a tap-shopify with GraphQL endpoint support.
- The duckdb target got support for MotherDuck!
- A new tap-contentful variant.
- Prime quality tap-hubspot and target-hubspot connectors powered by Meltano.
- A target-mysql.
- A new tap for mailchimp, based on the Meltano SDK.
- A high quality GA4 tap, built on the Meltano SDK.
- A new tap-netsuite built on the Meltano SDK.
- A tap for the Theme Parks wiki.
- New contributor instructions, just in case! (Also read the updated contribute.md)
Meltano Related Content
A lot is happening around Meltano on the web. We’re selecting a few pieces here. Be sure to check them out.
- We published an article on how to make data ingestion work fast and robust!
Keep on being Meltastic! And if we missed your contribution to the community, just hit us up on Slack, LinkedIn, or wherever.