It is now possible to run connectivity tests on any tap created in the Singer ecosystem. Previously there was no way of running simple tests on the hundreds of non-Meltano SDK taps already in existence. Even better – this test can be set as a scheduled job, giving you early notice of issues affecting your production data pipelines.
Test your taps
As of Meltano v1.80.0, you can simply run meltano config tap-myawesometap test and Meltano will execute your tap, wait to see if the tap is able to produce records, and then exit with success or failure as soon as connectivity is confirmed. While this feature comes now comes built-in for newer SDK-based taps, there previously was no means of running standard tests overs the hundreds of non-SDK taps created by the community over the past many years.
Why is this a big deal?
In one word, the Singer ecosystem is BIG. On MeltanoHub alone we list 300+ connectors created by members of the Singer community. This is the largest list of connectors of any data movement (ELT/ETL) provider in the world. Some of these taps and targets are actively developed while others have been stable and unchanged for years with little or no maintenance applied. Now you can quickly confirm that your tap is working and properly configured before adding it to your date pipeline.
Great! So how do I add testing to my own Meltano data project?
If you are already using Meltano, this couldn’t be easier: whenever you are considering a new tap you’ve not used before and want to check that the tap “still works”, simply run meltano config tap-mynewtap test. This will give quick confirmation that connectivity is properly established, config is valid, and record data is able to be extracted from the source.
For an added bonus, you can also scheduled this test in your own environment, to provide early warning when credentials may have expired or other connectivity issues might block your production data pipelines.
I manage some taps. Can I use this in my tap’s CI/CD pipeline?
Yes! And there are a number of ways to set this up. The easiest method is to first add a meltano.yml file to your repo which lists the accepted settings and their types, and then simply add a CI step that runs the meltano config ... test command from your CI/CD pipeline. As an added bonus, you can then directly set your tap’s config using environment variables for any secrets or dynamic settings. This does not requiring changing or refactoring your tap in any way, and the test is fast! The test succeeds as soon as the first record is confirmed to be retrieved.
This test won’t catch every type of error, but is a quick and easy way to ensure the code compiles and runs. The new testability adds significant value with a relatively minor investment of your time. Our suggested best practice is to set this up on a schedule so that early warning can be provided if the upstream auth API ever changes.
Building Meltano with and for the community
Last but certainly not least, a big thanks to Reuben Frankel for contributing this exciting new feature! If you are interested in contributing, please join us in Slack or come to our weekly Office Hours to discuss how you can get involved. If you are curious about building your own taps or porting an existing one to the SDK, please also consider joining our Tap-toberfest weeklong event, starting October 14th.