Today we are excited to announce the latest v0.3.0 version of the Meltano SDK, with two game-changing new features: Targets and Stream Maps.
Build-Your-Own Singer Target
Singer “targets” are flexible, composable data loaders which can be leveraged in any Meltano pipeline or any other Singer-compatible orchestrator. For taps and now also for targets, the SDK manages the internals of generating and managing the pipeline so developers can focus on how they want data to appear in their destination. As with taps, our goal for SDK-based targets is that developers can write (and maintain!) 70% less code than they would versus building from scratch. And unlike building taps and targets from scratch, building on the SDK gives developers a constant stream of bug fixes, performance improvements, and new features built directly in the SDK, centrally managed by Meltano, Meltano users, and the broader Singer community.
As of today, targets can be implemented with either a RecordSink class, for destinations which prefer singleton writes, or a BatchSink class, for targets which are more efficiently loaded in bulk. Simply download the template, select the best paradigm for your destination, and you’re on your way! For inspiration, you can also check out target-athena for a full working example. More code samples will be coming soon!
User-configured transforms, filters, and aliasing
With the new Stream Maps feature in today’s release, we’re adding powerful transformation, filtering, and aliasing capabilities for both taps and targets. Users can now add inline mapping features directly in the config of any tap or target plugin built on v0.3.0 or greater. Use cases for stream maps include:
- stream and property aliasing
- duplication and bifurcation of streams
- hashing or anonymizing PII/PHI
- row-level filtering
- suppression of streams or stream properties
- and more…
These features are available automatically for users of taps and targets built on the SDK version >=0.3.x. However, there are already hundreds of taps and targets out there which were built prior to the SDK launch. To support stream map features for non-SDK taps and targets, Meltano will soon also be building and releasing a standalone stream map plugin, bringing the above stream map capabilities to all Singer taps and targets, including all of the 275+ taps and targets listed today on MeltanoHub.
A broad vision for ELTP, including “Reverse ETL”
The “ELT” DataOps vision championed by Meltano and others has now thoroughly supplanted older “ETL” approaches as industry best practice. Now, with Stream Maps and Targets released, we are adding “Publish” to the canonical data pipeline lexicon, embracing a more end-to-end “ELTP” approach by adding capabilities for the “Publish” step, sometimes called “reverse ETL”.
With the dual release of Targets and Stream Maps in v0.3.0, we’re making it easier than ever to publish data to non-traditional destinations, such as SaaS targets and other API-based endpoints. We look forward to seeing how developers apply these tools together to push the boundaries beyond what is possible with traditional data pipelines.
What else is new?
The list below (copied from the changelog) covers all of the changes made to the SDK since our last v0.2.0 release on May 20.
Added
- Added Stream Map feature for inline transformation and filtering capabilities (#63, !4, !92, !103)
- Added Target SDK, components and templates for building Singer targets (#96, !4)
Removed
- Removed methods deprecated in v0.2.0:
get_partition_state()andget_stream_or_partition_state(). Affected developers should replace these with references toget_context_state(). (#152, !107)
But wait… there’s more
As excited as we are about this latest release of the SDK, there’s still more good news just around the corner! We hope you will join us for a very special office hours this Wednesday, June 30th, where we will introduce our newest team members and share some exciting updates on the future of Meltano.
As always, you can find us on Slack in the #singer-sdk channel, and/or join the conversation in weekly #office-hours.
Hope to see you soon!
-AJ