Meltano

Now Available: Meltano v1.76.0

Today, we are excited to release Meltano version 1.76.0, which:

  • resolves an issue with excessive “cache_ok” warning messages logged while running meltano elt
  • fixes a bug where catalog discovery would fail when using plugins with no pip_url set, for example with SDK-based taps
  • fixes a problem in plugin definitions for tap-chargebee, tap-intacct and tap-quickbooks, and enables stream selection features for these taps

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

This post describes all Meltano updates since the v1.75.0 release on May 28. For a full history of recent updates, please see the Meltano changelog.

And lastly, in case you missed it, don’t forget to check out the new MeltanoHub site for discovering taps, targets, and other Meltano plugins, and our new SDK documentation site, which makes it easier than ever to build high-quality custom data connectors.

Demo Day Recap: 2021-06-04

It’s the first Fri-yay of June, which means it’s time for another Demo Day! If you missed it, the recording of the livestream is available on the Meltano YouTube channel.

Taylor kicked off Demo Day by recapping our launch of MeltanoHub for Singer on Monday. He then went through a detailed demo of adding tap-exacttarget to the Hub.

Next we had community member Pablo Seibelt share how he used the Meltano Tap SDK to build a tap for Decentraland to pull data from thegraph.com.

Lastly, Ruslan Yangazitov from Applied Labs shared more about the work they’ve been doing with Singerly which is a hosted platform for Meltano and Airflow. They’ve recently added the ability to add custom taps directly from the interface and the ability to manage billing. Douwe also talked about how the Meltano team loves seeing efforts like Singerly because it helps the entire Singer ecosystem to grow and aligns well with our embrace of the Singer community.

Overall, this was a fantastic Demo Day and we’re very appreciative of community members who take the time to share their work. We’re always listening to feedback and looking for ways to improve, so please contribute to any issue within the main project or chat with us on Slack if that’s more your style. Thank you to everyone who joined the livestream!

Join the conversation!

Our next Meltano Demo Day is on Friday, June 18 and we have our next office hours session on Wednesday, June 9. We hope to see you there!

Office Hours Recap: 2021-06-02

In case you missed it, we covered three big topics this week in office hours: Meltano Hub launch (which we’ll demo officially in Friday’s Demo Day), the new Stream Maps capability we are adding to the Singer SDK and Singer ecosystem, and our work on a native Athena target for Meltano and dbt. The full video is below.

Click through to YouTube description for topics and timestamps.

New Stream Maps Feature

The bulk of time this week was dedicated to discussing the new draft spec for our Stream Maps capability. This new capability adds a variety of new inline stream transformation capabilities, including: data encryption, PII obfuscation, stream aliasing, stream bifurcation, stream filtering, and more. This is an extremely powerful feature, and we believe the entire Singer community can benefit from these new capabilities.

When the feature launches later this month, it will come out-of-box on all SDK-based taps and targets, and we will also launch a custom plugin which can be placed between any tap and target, even legacy taps and targets which were not built on the SDK.

Join the conversation!

To learn more about these features, or to provide input, please find us on Slack or drop a comment into any of your favorite issues or merge requests.

See you soon!

AJ

MeltanoHub for Singer launches with over 200 Taps and Targets!

The Meltano team is excited to announce the official launch of MeltanoHub!

MeltanoHub, or the Hub for short, is the best place to discover Singer Taps and Targets and all plugins supported out-of-the-box by Meltano. 

There are over 200 taps and targets on MeltanoHub, making it the most comprehensive catalog of open source data connectors anywhere.

Since pivoting to focus on ELT last year, we’ve done many things to lift up the Singer ecosystem. We’ve made Meltano a fantastic way to run taps and targets and deploy them in a production environment. We’ve created the SDK for Taps (and soon for Targets!) to make it as easy as possible to build high quality connectors without needing to understand the full Singer specification (though if you’re interested in the details of the specification, we’ve worked to simplify it so it’s easier to understand everything).

And now we’ve built MeltanoHub to solve one of the last challenges facing the Singer ecosystem: discoverability.

One of the challenges for existing Singer resources, such as the singer-io GitHub namespace or the Singer.io site, is that they aren’t regularly updated and don’t showcase the activity of the many participants in the ecosystem. In reality, there are over 100 existing taps not listed on the canonical Singer repositories and of the ones that are listed, there are more active and updated forks that aren’t surfaced to the community. What this means is that it’s not clear to anyone, let alone participants in the ecosystem, that Singer has the most connectors of any data integration tool (open or closed source)! These sites also don’t take the opportunity to showcase the dozens of active consultancies and companies who build Singer connectors or use them within their own products. 

We believe the launch of MeltanoHub for Singer makes these great parts of the Singer ecosystem evident while simultaneously elevating the ecosystem to the next level of usefulness.

We had three main goals with the launch of MeltanoHub:

  1. Showcase the majority of connectors that already exist within the Singer ecosystem
  2. Make the directory of taps and targets better than what exists today
  3. Enable any person or organization to build on top of the large library of Singer connectors

We believe we’ve achieved each of these goals with this launch. By pulling data from GitHub we’ve been able to list 201 taps, but we know there’s more we have yet to add! For each connector we’ve curated the best forks and repositories, added metadata, and stored the information in a YAML file (validated via a JSON Schema) that can be iterated on in collaboration with the community. We’re also pulling metrics from GitHub for every tap so users can get a sense of the quality and activity of a given variant.

All of this work is open to everyone via the website and repository, but also via an API. We have endpoints that return a JSON listing of all taps and targets on the site. Meltano itself will be driven by this data and we hope others use this data to build their own catalog of taps and targets within their products. All of these features are detailed more in the Hub docs and we’ll share in future blog posts more about the engineering behind MeltanoHub.

The Future of Singer is Bright

The Meltano team will continue to lift up the Singer ecosystem through MeltanoHub, the SDK for Taps and Targets, and via Meltano itself.  We believe in a truly decentralized open source ecosystem and MeltanoHub reflects that. In the coming months we have many planned iterations for the Hub including features like adding more metrics to each connector page, adding more details about a connector’s capabilities and settings, increased automation around quality metrics, adding more taps as supported out-of-the-box by Meltano, and deeper integration between the SDK and the Hub.

We think you’ll find value in the Hub, but we want to hear from you! If you see anything that can be improved, make an issue in the project or, better yet, make a merge request with the change! Also, join the community on Slack where you can keep up to date on everything Singer and Meltano.

Now Available: Meltano v1.75.0

Today, we are excited to release Meltano version 1.75.0, which:

  • allows meltano commands to be used from a subdirectory of a project (thanks Charles Julian Knight!)
  • adds support for already-installed taps and targets by making pip_url optional when adding custom plugins
  • allows non-pip plugins to be declared using relative executable paths and executable programs already on the user’s path

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

This post describes updates since the v1.74.0 release on May 11. For a full history of recent updates, please see the Meltano changelog.

Demo Day Recap: 2021-05-21

Another Friday, another fantastic Demo Day! If you missed it, the recording of the livestream is available on the Meltano YouTube channel.

Taylor kicked off the session by talking more about our exciting announcement: the MeltanoHub is now the most comprehensive source for Singer Taps. It lists 184 taps and we expect to add more over the coming weeks and months.

AJ followed up this announcement by diving into some of the work that went into getting to this number. The work he’s been doing touches all parts of Meltano and Singer:

We want this project to be a great example of how to use Meltano for production use cases. We’ll continue to dogfood the tool and keep this as a representative method of structuring and organizing a Meltano project.

Next we had Daigo Tanaka from ANELEN talk about his consulting company and Handoff, the open source tool that he wrote for his clients. ANELEN has multiple taps on the MeltanoHub and is now also listed on the partners page for Meltano.

Lastly, we had Florian from Kuwala share some about their open source location data integration project.

This was a great Demo Day and we’re really appreciative of community members who take the time to share their work. We’re always listening to feedback and looking for ways to improve, so please contribute to any issue within the main project or chat with us on Slack if that’s more your style. Thank you to everyone who joined the livestream!

Join the conversation!

Our next Meltano Demo Day is on Friday, June 4 and we have our next office hours session on Wednesday, May 26. We hope to see you there!

Singer SDK v0.2.0 is now available!

Today we are excited to announce the latest version of Singer SDK for taps! In this release, we are adding the ability to create Parent-Child stream relationships, which was the most-requested feature by developers using the SDK.

Where previously, tap developers needed to create complex nested loops and error checking, this new SDK feature quickly and easily define relationships between parent and child streams. The SDK automatically takes care of orchestration, error handling, and state tracking. Check out the updated docs for instructions and sample code to get you started.

What else is new?

The list below (coped from the changelog) covers all of the changes made to the SDK since our last v0.1.6 release on May 14.

Added

  • Added support for parent-child streams (#97, !79)
  • Added support for configurable metrics logging (#91, !79)
  • Added ability to use fewer state bookmarks by setting Stream.state_partitioning_keys to a subset of available context keys (!79)

Changed

  • Renamed the optional partition dictionary arg in method signatures to the more generic context (!79)

Deprecated

  • The methods Stream.get_partition_state() and Stream.get_stream_or_partition_state() have been deprecated in favor of the new and simpler get_context_state() (!79)

Improved

  • Code coverage is now tracked and available as a tool for SDK contributors to further improve overall stability and help prioritize unit test development. (#39, !89)

Come see the SDK in action

Tomorrow in our fortnightly Demo Day, we’ll be demoing an end-to-end Meltano implementation which utilizes the new Parent-Child streams capability. We’ll also demo an AWS Athena loader built with the Target SDK Preview and our Athena dbt transformation pipeline running natively against S3 files.

As always, you can find us on Slack in the singer-sdk channel and/or join the conversation in weekly #office-hours.

Hope to see you soon!

Singer SDK v0.1.6 is now available!

I am excited to announce the latest version of the Singer SDK. With this release we’ve focused on bug fixes and stability improvements that will continue to make the tap development process better than ever.

Here’s the full list of fixes and updates in this release:

  • Resolved excessive logging during selection filtering. (!83)
  • Resolved issue where deselected sub-fields were incorrectly included in stream records. (Thanks, Alex Levene!)
  • Added improved type hints for developers, including mypy code compliance for improved stability. (!86)

If you’ve already built a tap with the SDK, simply run poetry update singer-sdk to import the latest SDK version for your tap.

Two big features now in preview

While this release focused on stability improvements, we’ve also been simultaneous building out the two most requested features: our Target SDK and Parent-Child Streams and we have an early preview now available for both.

If you would like to take either of these for a spin, please check out the preview of the Target SDK here and the Parent-Child Streams here. Now is the perfect time to checkout these upcoming features and provide your feedback.

Join the conversation

For more information or to join the conversation, find us in the #singer-sdk channel on Slack! We also have #office-hours every Wednesday and #demo-day on fortnightly Fridays!

Building Meltano in Public: Bimonthly Recap

Last week, it was once again our turn to host a GitLab Group Conversation (a publicly live streamed Q&A on the GitLab Unfiltered YouTube channel) on Meltano!

We used the opportunity to share a recap of:

If you’re curious, check out the presentation on Google Slides and the Q&A on YouTube. The presentation content is also reproduced below, as is an embedded video of the Q&A!

Group Conversation Presentation

4 releases since the last GC (2021-03-15)

  1. V1.71.0 – Adds support for utility plugin type, added out of the box support for tap-zoom, fixes a bug to prevent false positive matches in passwords 
  1. V1.72.0 –  Added support for shortcut commands to invoke,  Added support for sqlfluff utility for linting SQL transforms,  Added mashey variant of tap-slack,  Added documentation for using a custom Python Package Index (PyPi),  Added transferwise variant of target-redshift. Fixes an error when when target process fails before tap. 
  1. V1.73.0 – Added twilio-labs variant of tap-zendesk, Speed up meltano install by installing plugins in parallel, Added support for setting kind in settings prompt when using meltano add –custom
  1. V1.74.0 – Add meltano remove command

SDK for Singer Taps Launched with 5 Subsequent Releases!

  1. v0.1.0 – SDK Launched! Announcement post: https://meltano.com/blog/2021/04/05/meltano-launches-v0-1-0-of-the-singer-tap-sdk/ 
  1. v0.1.1 –  Added ‘admin_name’ field in cookiecutter, streamline poetry setup, Added meltano integration and testing options , Added new cookiecutter .sh script to ease testing during development
  1. v0.1.2 – Fixes bug in Stream.get_starting_timestamp() using incorrect state key
  1. v0.1.3 – Large update to add support for unsorted streams
  1. v0.1.4 – Adds support for GraphQL variables and easy stream selection
  2. v0.1.5 – Resolved tap failure when a sorted stream has non-unique replication keys

22 Recent contributions by 9 community members

Done

  1. Meltano: Support for a generic plugin type for arbitrary python cli plugins by Charles Julian Knight (FIXD)
  2. Meltano: Add documentation for using a custom PyPI package by Benjamin Maquet (Twilio)
  3. Meltano: Updated the database_uri documentation for target-postgres by Daniel Walker (Matatika)
  4. Meltano: Adding pipelinewise target-redshift by Ingo Klose (b.telligent)
  5. Meltano: Add support for shortcut commands for plugins by Charles Julian Knight (FIXD)   
  6. Meltano: Disallow parallel pipelines by default by Charles Julian Knight (FIXD)
  7. Meltano: Add tap-zendesk variant by Benjamin Maquet (Twilio)
  8. Meltano: Configure kind of setting via prompt when adding a custom plugin by Reuben Frankel (Matatika)
  9. Meltano: Add `meltano remove <type> <name>` command to remove a plugin  by Reuben Frankel (Matatika)
  10. Singer-sdk: Fix reading catalog from JSON file by Edgar R. Mondragón (SpotOn)
  11. Singer-sdk: Fixed bug in STATE handling by Ken Payne (Tails.com)
  12. Singer-sdk: Look only for valid plugin settings in environment variables by Edgar R. Mondragón (SpotOn)
  13. Singer-sdk: Fixed issue reading from JSON file by Edgar R. Mondragón (SpotOn)
  14. Singer-sdk: Added default value support for plugin configs by Ken Payne (Tails.com)
  15. Singer-sdk: Fixed a problem where CI pipelines would fail when run from a fork by Derek Visch (Auto IDM)
  16. Singer-sdk: Fixed a scenario where unsorted incremental streams would generate incorrect STATE bookmarks by Egi Gjevori (y42)

In development

  1. Meltano: Update default dbt version to 0.18.2 by Andrew Stewart
  2. Meltano: Add fork of target-csv as new default variant by Hassan Syyid (Hotglue)
  3. Singer-sdk:Add built-in support for FAST_SYNC spec by Ken Payne (Tails.com)
  4. Meltano: Auto-generate dbt sources.yml from extractor/loader schema by Andrew Stewart
  5. Singer-sdk: Support defining configuration and stream schemas using Pydantic by Edgar R. Mondragón (SpotOn)
  6. Meltano: Allow Meltano to be run from subdirectories by Charles Julian Knight (FIXD)

Other exciting recent and ongoing developments

Recent and projected weekly Slack activity

This week’s priorities

Upcoming priorities

Milestones issue board

Epics:

Group Conversation Q&A

Now Available: Meltano v1.74.0

Today, we are excited to release Meltano version 1.74.0, which:

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

This post describes updates since the v1.73.0 release on April 29. For a full history of recent updates, please see the Meltano changelog.