You’re likely using Meltano to integrate data from various sources into one database, data lake, or data warehouse destination. But Meltano can be used for any pipeline that has one step to generate records (not limited to pulling from APIs or DBs), zero or more to modify them on the fly (not limited to renaming or hashing columns), and one to do something with those records (not limited to pushing into a DB). Many processes you may want to automated take that shape.
Take the Tidbyt, a little 64×32 display that cycles through apps (little widgets showing dynamic content and animations) that you can add through its mobile app, including ones built by the community using its Pixlet SDK. The Tidbyt device does not run apps directly; it depends on a server to periodically run apps and push the resulting images to the device. Apps installed through the mobile app run on Tidbyt’s official app server. For security reasons, these apps are implemented in a limited Python-like language named Starlark that supports only basic HTTP requests to the public internet, which means some awesome app ideas that depend on complex network protocols and local network resources (like IoT devices in your home) are impossible to implement in community apps.
That looks like a pipeline:
- run an app to generate an image,
- then send it to a Tidbyt device,
- and do this as often as necessary to keep the app on the Tidbyt up to date.
In other words:
- a Singer tap that runs an app,
- a Singer target that pushes it to the Tidbyt over the API,
- Airflow to run this on a schedule,
- and a Meltano project to package them all together into an easily deployable Docker image.
That’s exactly what our CEO Douwe built as a side-project: tap-pixlet, target-tidbyt, and a Meltano project that ties them together named Pixbyt: “a self-hosted Tidbyt app server for advanced apps”. Tidbyt users can fork the project, add their apps, and spin it up on their homelab, without knowing that it’s Meltano and Singer doing the heavy lifting!
Did you find any novel uses for Meltano? We’d love to hear from you – just hit us up on Slack or any of our other social media.