TL;DR: Thanks to an amazing community contribution by Alex Butler at Harness.io, Meltano now supports over 200 Airbyte connectors in addition to over 300 Singer taps, making it the EL tool with the largest connector library in the world.
The best part about open source software is that great ideas and contributions come from outside the core team. The Meltano community is no exception.
Alex has been a frequent contributor to open source projects. He’s created Meltano Power User, dbt Osmosis, and made numerous contributions to MeltanoHub, the Meltano Singer SDK, dbt, and other projects.
His latest contribution is something we’re very excited about: tap-airbyte-wrapper.
What If?
This contribution started with a question from him – would it be possible to run other connectors with Meltano? And more specifically – would it be possible to run Airbyte connectors on Meltano?
The short answer is yes – and without too much trouble!
At its core, tap-airbyte-wrapper calls the Docker image for the connector and does the appropriate translation for the config, catalog, and state inputs. When the container outputs Airbyte messages, it translates those into the Singer specification and sends them on their way to your loader.
With the Meltano Singer SDK, it took less than 1000 lines of code to make every Airbyte source compatible with the Singer specification and therefore runnable within Meltano.
Get Running
To make running these additional extractors as easy and joyful as possible, we’ve added the majority of Airbyte sources to MeltanoHub. This makes it so that anyone can simply run
meltano add extractor <tap-name>to add the connector to their Meltano project.
This integration also works with GitHub codespaces! If you have a GitHub account, you can test out a connector yourself by opening a new Meltano codespace. If you follow the README there you’ll have a pipeline running in less than 5 minutes and from there you can try installing any Airbyte connector!
As you can see, we’re very excited by this community contribution. To help the community, we’ve compiled a short FAQ page with common questions you might have about this integration.
Office Hours Video Intro by Alex
If you’re interested in the details, Alex shared the wrapper with an example in the video below!
We’d love your feedback on this as well! Please join our Slack community or open an issue if you have questions or just want to chat with the team!