Our Vision for The Future of Meltano

Recently, the Core Team met to discuss our visions of what Meltano could become over the coming years. It was a wonderful, multi-hour discussion and all of us left feeling excited about the potential and eager to manifest that future. In this post we wanted to share the highlights of what we talked about and how we think about the future of Meltano.

Constant Commitment to Open Source and the Data Community

While we have a lot of features on the roadmap, one of the things we talked about is the importance of our values and the community. To that end, we recognized that having a healthy open source ecosystem and a strong data community are going to be key drivers of our success over the long term. 

One part of our mission with Meltano is to level-up the Data Profession. This means recognizing the value and competitive advantage that Data Professionals and Data Teams bring to an organization. Many Data Teams were built as an afterthought when an organization realized it “needed to do something with its data”. The state of tooling for Data Teams reflects that reality: it is disjointed, challenging to use, and squanders a huge opportunity for organizations everywhere.

We believe that building open-source software and fostering and supporting OSS within the data ecosystem is essential to achieving our full mission. OSS has elevated the best practices within software development and we know it can have the same effect on the Data Profession. We believe Singer will be the de facto standard for any data integration challenge. We also believe that Open Source will win many parts of the data stack. We want to integrate with and support those tools as well.

We also want the Meltano community to support all data professionals, not just Data Engineers. We aim to provide training and education for the entire data lifecycle, including for Meltano specifically. As the data profession grows into fully embracing DataOps we want to help people onboard to this new and better way of working. With so many open source tools available it can be daunting to know where or how to get started. We aim to make the tooling as understandable as possible while giving people the confidence to jump in and get working with a thoughtful, curated stack that supports modern best practices.

We recognize the fantastic responsibility of stewarding an open source project and community. We aim to do what’s best for Meltano and the larger data community so that we can sustainably build a great tool for years to come.

In 5 years – Be the One-Stop Shop for Data Teams

In the long-term we aim to go big. We believe that Meltano can become the one-stop shop for the majority of data projects. Here are some of the key things we want to see in Meltano less than 5 years from now.

Deep First Principles Integration – Issue #2667

We believe it will be critical to have DevOps best practices in the DNA of the product. This means having a strong notion of version control, CI/CD, and the development experience. It should be easy to have disposable environments for the entire stack (think about Snowflake zero copy clones). We believe these best practices to be important because DevOps tooling enables software teams to move faster with more confidence, reliability, and security – Data Teams will need to move faster as well since high-performing Data Teams are becoming more critical to the success of all businesses.

Fantastic User Interface – Epic #78

While we are building Meltano CLI-first, we know it will be essential to have a great UI for the tool as well. We believe that building for the command line first and growing from there will lead to a better experience and growth path for the product. But for us to be able to compete and bring in a whole new generation of data professionals we will need an excellent user experience with a well designed UI.

Enterprise Ready – Issue #2667

We want Meltano to be a long-lived project with a sustainable business supporting it. To do that we need to sell to large enterprises. This means thinking about features needed by these organizations before we’re ready to sell to them. We believe we can do this in a way that is supportive of the larger data community by following a buyer-based open core business model where only those features that are more relevant for managers and up are what we charge for. We will also want to have a SaaS and a Self-Hosted option that are equivalent to each other.

Go Big!

Meltano has a big future ahead of it and we’re confident in our ability to manifest it. But how do we get there? One iteration at a time. To that end, we also talked about where we want to be within the next year.

Meltano by 2022-H1

By next year we want to have all of the following done (and more, of course!).

Modernized and Enhanced Singer Ecosystem – Epics #87, #85, #80, & #88

Our work with the Singer Software Development Kit (SDK) is well underway and it will become the de facto way to build new taps and targets. Along with the SDK, we will soon be building the SingerHub which will be the easiest way to find working and up-to-date taps and targets. 

These investments into the Singer community will continue with support and enhancement of the protocol itself. We’ve already released a first iteration of our interpretation of the Singer spec. We also plan to update and support optional extensions such as `ACTIVATE_VERSION` and `BATCH` (aka Fast Sync) message types. We’re also thinking about how to have a generic data plugin architecture for Singer so that data can be obfuscated or dropped on the fly.

Enhance the Command Line and Configuration Experience

Because we’re building Meltano with a CLI-first focus, we want to make working on the command line within your Meltano project even better. Bringing better decentralization and composability to a project is a part of that goal. This means making it easier to simplify configuration and have it spread out across your Meltano project by being able to split up your `meltano.yml` file and even import other projects into one another. Since a key part of the promise of Meltano is that it will integrate with many tools, being able to integrate with itself will be a core concept. Making Meltano git-aware is something we’re eager to add as well since it will help with the development and deployment of Meltano.

Metadata, Monitoring, Observability – Epic #93

Meltano as a product sits in a unique position because it is at the start of the lifecyle for any data project. Part of the promise of an all-in-one tool is that it makes it possible to easily understand that status of data anywhere along the value chain. We want to make metadata a native part of the product and build this capability early.

Great Deployment Experience – Epic #28

Deploying Meltano is a critical part of making it usable. We want to make it as easy as possible to deploy Meltano in any environment. To do this we’re going to enhance our dogfooding of the product as well as simplify the deployment process as much as possible. Great documentation will be a key piece of this too.

Full Open Source End-to-End Solution

Out of the box we want to have everything you’d need to get a modern data stack up and running. This means everything from data integration to orchestration, warehousing, transformation, operational analytics (Reverse ETL, if you like), and reporting. We’re going to do this by integrating with tools like Dagster, Prefect, Superset, and Jupyter Notebooks, but also by having support for a modern analytic datastore such as Spark, Clickhouse, or DuckDB. We’re also going to dramatically improve our dbt integration by doing things like autogenerating sources.yml files and adding native support for testing. Having this end-to-end stack in one place will enable unprecedented coordination between the different tools meaning we’ll be in a great place to achieve our long term vision.

The Future is Bright

As you can see, we’re all extremely excited about this space. We want to build a tool for Data Professionals so that organizations can manage and access their data more effectively so they can enable better decisions. We want to level-up the status of the Data Profession so that those who work in it are respected for the true value they can bring. We believe that the data community is growing more technical while the value of data within organizations is becoming more valuable. 

We hope to support the community through the Meltano product but also via the open source ecosystem too. Meltano will be driven by what the community wants and needs, so we need your active involvement! Please collaborate with us on issues, in Slack, and on Twitter for how we can make the product and the community better! 

Office Hours Recap: 2021-04-07

This week in Meltano office hours we discussed how the newly released Singer SDK can dramatically speed up the development of new Singer Taps, as well as our newly launched Singer Spec documentation, which make it easier than ever to understand exactly how the open source Singer framework actually works behind the scenes.

You can watch the full livestreamed session below or on YouTube. Check the annotations in the video description if you want to jump straight to a specific topic.

Meltano Office Hours – 2021-04-07

We hold office hours every Wednesday, 9AM Pacific / 16:00 UTC. For more info, including the link to each week’s Zoom, check join the #office-hours channel in Slack. If you are interested in showing off your latest work, or seeing what others have built, please also join us this Friday for Demo Day!

Hope to see you soon!


A Simplified Take on the Singer Spec

The Meltano team is excited to share a recent update we made to our documentation of the Singer specification: we’ve created an interpretation of the spec that is easier to understand and includes sections for different audiences. 

Part of our mission with Meltano is to embrace the Singer community by building tools and resources that lift-up the entire ecosystem. Our launch of the Tap SDK on Monday is one example of that effort. 

Our Singer Spec interpretation is the next example. One of the challenges we saw for new people to the Singer ecosystem is that it can be difficult to “grok” everything that’s going on with the spec as it’s written on on the singer-io GitHub group. We identified this as one of the challenges slowing down the community and took it upon ourselves to write a version that is more focused on the core of the spec and is free of heavily-intertwined, vendor-specific information.

To be clear, we have no intention of replacing the Singer spec. This interpretation is meant to consolidate some of aspects of the spec that were spread across multiple pages and bring them together into a single resource for ease of use. 

Our intention going forward is to move this out of the Meltano website and onto the SingerHub, once it’s launched, so that it’s even clearer that the spec isn’t tied to any specific company or project. 

We hope you find it valuable! If you have any suggestions please submit an issue or merge request, or join us on Slack to chat more about Singer, Meltano, and open source data integration.

Special thank you Derek Visch and Charles Julian Knight who contributed feedback to an early draft of the document.

Meltano Launches v0.1.0 of the Meltano SDK for Singer Taps

Today the Meltano team is excited to announce a milestone for the Singer community: the v0.1.0 launch of our Singer Tap Software Development Kit! The SDK is a framework that makes it easier than ever to build high quality data extractors, aka taps. With the SDK, tap developers can take full advantage of the Singer spec without being an expert on it, while enabling them to focus on the code unique to the API or database they are extracting data from.

This is one of the many things we’re working on in the coming months to lift up the entire Singer and data integration ecosystem. In addition to continued development of easy-to-use SDKs, we’re also working to simplify the spec so it’s easier to understand for people new to the community. We’re also building a MeltanoHub for Singer so developers can easily find, use, and contribute to high quality taps and targets. There’s also been huge interest from the community around extending the Singer protocol and we expect our efforts with the SDK and within the ecosystem to make it easy for the spec to grow and evolve.

Genesis of the SDK

As you can see, we’re bullish on the potential of Singer and its community, but we recognized the real challenges of building and maintaining high quality taps and targets. We saw that standardization across taps could be dramatically improved – some would implement all parts of the Singer spec while others did the bare minimum. We also saw how challenging it was to build a new tap with all of the great features of the protocol. 

AJ Steers, who joined the Meltano team last month, saw these problems too and started work on the SDK as a community member. As a consultant, AJ has experienced the challenge of building taps that implemented all parts of the Singer spec without any sort of scaffolding to help him. “Optional” features such as incremental replication and stream/property selection were difficult to add.

The SDK solves all of these problems. Developers now don’t have to become an expert in the spec to write a high quality tap. They can focus on the code that’s unique to the API or database and will get all of the good stuff for free. 

Meltano SDK In the Wild

Our goal with the SDK is to decrease the amount of code developers have to write by over 70%. To that end, we have a cookiecutter template which will ask you a few questions on the command line:

> cookiecutter https://gitlab.com/meltano/singer-sdk --directory="cookiecutter/tap-template"
source_name [MySourceName]: mytap
tap_id [tap-mytap]: 
library_name [mytap]:
Select stream_type:
1 - REST
2 - GraphQL
3 - Other
Choose from 1, 2, 3 [1]: 
Select auth_method:
1 - Simple
2 - OAuth2
3 - JWT
4 - Custom or N/A
Choose from 1, 2, 3, 4 [1]: 

And then within the code we’ve marked multiple #TODOs to highlight where you need to add the custom code for your source. It’s that easy!

Even before this official release, we had many community members interested in the project who started to use it before it was “ready”.

Edgar Ramirez from SpotOn is developing a tap for the Confluence Content API. Edgar said even before the v0.1.0 release,  “I’m focusing on extracting data from REST APIs at the moment and it’s amazing how quickly you can get your tap to output Singer spec messages. [It’s just] a few lines of code, 90% of which deal with idiosyncratic pagination and authentication, the rest are just stream (name, schema, primary keys) and tap (config schema) declarations”.

Derek Visch from AutoIDM needed a quick proof of concept to show he could make a connection work between BambooHR and ActiveDirectory. “Using the SDK, I had BambooHR data coming through in 2 hours, and that’s with no Python experience.”

John Timeus from Slalom developed a tap for PowerBI Metadata and even our own Douwe Maan shared in our first Demo Day the tap he built to pull data from investing.com.

But don’t just take their word for it – check out our dev guide to get starting making your own tap!

The Future of the SDK

v0.1.0 of the SDK is just the beginning. While it supports the basics you’d expect such as data sync, catalog discovery, and bookmark tracking, we have a plethora of features we want to add to the SDK. This includes supporting stream/property selection, database type streams, auto-generation of documentation, and handling unsorted streams. We aim to have the SDK support everything the Singer spec currently allows for, but we also want to build support for extensions on the spec including the `ACTIVATE_VERSION` and `BATCH` message types. You can see these issues and more in the issue tracker

We’re also planning to expand the SDK to include Targets as well as for plugins that can ingest and export Singer valid data. This latter use case enables on-the-fly stream transformations like hashing or aggregation.

We hope you can spend some time building taps with the SDK! Our expectation is that all existing taps will eventually be ported over to the SDK. When this happens, the community will benefit from taps that are easier to maintain, are more consistent across different sources, are under a more permissive open source license, and can immediately benefit from enhancements to the SDK and Singer spec.

What’s Next

Keep an eye out later this week for two more blog posts: the next one will be about the updates we’re making to the Singer spec to make it more understandable and the other will be about our long-term vision for Meltano.

On Wednesday April 7th we’ll have our weekly Office Hours which will be focused on the SDK and on April 9th we’ll have our Demo Day where community members and the core team share what they’ve been working on. In the meantime, join us on Slack to chat more about the project, or file an issue if you have feature ideas or run into any problems. 

Happy building!

Meltano is Special: AJ Steers explains why he’s all-in on Meltano

Earlier this month, I joined the Meltano team full time with two ambitious goals: (1) redefine how organizations think about their approaches to data projects and (2) make this next-gen approach to data freely available to everyone, regardless of budget or the size of the data. From hobbyists working on pet projects, to multi-billion dollar corporations: I believe everyone can benefit from Meltano’s approach.

This post will explain why I decided to join Meltano, why I think DataOps is critical to a team’s success, and what I think is so unique about Meltano in particular.

ETL is dead; Long live EL and T!

To start off, one of the biggest reasons I joined Meltano is because I think transitioning to the new DataOps approach is exciting, and I want to do my part to make transitioning to DataOps a positive, joyful experience. If your data team has not taken the plunge yet, the plunge is coming and you should be getting ready! If you work in data, it is likely some time within the next 12 months, someone is going to excitedly come ask you about DevOps, it’s younger cousin “DataOps”, and CI/CD: they’re going to ask you if your data team has taken that plunge into the world of DataOps or if you are still using the old-style traditional ETL tools. I’m here to tell you life is much nicer here on the other side.

Addressing two important questions about the future of data teams

You might be in the data space yourself and you might be skeptical of the direction that tools like Meltano and dbt are moving in. Or maybe DevOps, git, and CI/CD seem like too much to learn. I want to address two of these concerns of these before I go further.

Objection #1: My data team aren’t coders. Do we really need git and CI/CD?

First of all, I know this is going to sound controversial, but your data team are already coders. They just might not yet have the tools they need to support them writing good code. In our own way, we are all coders whether we write code directly or use a tool to write the code for us.

As long as two or more people are working together towards a common goal, there will always be merge conflicts to resolve and code reviews to perform. Have you ever emailed a coworker a copy of a word document, only for both of you to make changes and then stumble through how to merge those changes back again into a single copy? That’s called a “merge conflict” – resolving it made you a coder! Have you enabled “track changes” on a document so someone else could review those changes one at a time? That was a code review and in that moment, you were a coder! These are exactly the types of source control problems which benefit from git and CI/CD. Historically, however, very few of us in the data community have had access to tools that managed our code effectively. That is, until now.

For anyone still nervous about git or “coding”, please consider checking out the Git and GitHub for Poets series. It’s clever and entertaining and does a great job demystifying all the DevOps technical jargon.

Objection #2: But isn’t coding hard? What about the move towards “no-code” solutions?

I want to address this point because I hear this question often in the data space. While we all want to be writing less code, more elegant code, and more efficient code, we can’t get away from code altogether; we can only invent simpler and more accessible languages to deal with it, and then build great tools on top of those languages. The Meltano UI, for instance, literally writes EL code for us. Afterwards, we’re still free to modify the code ourselves if we wish, track changes, and even undo changes.

Rest assured: the languages you learn when building a Meltano project are not going away anytime soon. These are the same languages that I’d bet on my 5 year old son learning in high school ten years from now: SQL, YAML, and Python. These are not only among the most popular languages out there, but they are also among the easiest that one can learn today! (Yes, much easier than French! 😅)

Why I believe Meltano is special

As I think about why Meltano is special, there are three big reasons that come to mind: specialization, composition, and collaboration.

Meltano treats EL and T as distinct problem types (Specialization)

Meltano understands that the challenges with EL (“Extract-and-Load”) and T (“transformation”) are each unique. As such, Meltano handles each with a targeted best-in-class approach for each: the powerful Singer platform as our EL framework and the revolutionary open source tool dbt as our data transformation layer. Meltano itself is the cohesive solution for the entire end-to-end data project. Looking to the future, beyond EL (Singer) and T (dbt), Meltano will similarly integrate with other great tools like Superset for reporting and BI and Great Expectations for validation and governance.

Meltano brings it all together in one place (Composition)

Despite the distinct and unique challenges of each stage of EL and T, Meltano knows that data professionals want a single home for the entire data project, with clear transitions and handoff between each stage of the process. For instance, today Meltano recommends dbt transformation packages based upon the data sources in your project. There is plenty of room for us to continue expanding this functionality in the future.

Meltano is Pro-Team, because… DataOps! (Collaboration)

If you haven’t heard the word “DataOps” before, please allow me to formally introduce you: “DataOps” is simply “DevOps for Data” – and at the same time it is also so much more. The term “DevOps” was coined in 2009 as an agile-driven movement in software development, automating the “ops” part of development and putting source control systems to work automating those steps we don’t need to be spending time on manually. Deploying, packaging, testing, rolling back: all of those tasts are handled automatically by what we now call “CI/CD”. All we need to do is write some tests, use automation to continually retest those tests, and then finally we can trade our manual deployment headaches for a streamlined code review and approval flow. With DataOps, anyone can send a proposed code change, code owners can review that change, and the systems literally do the rest.

With DataOps and Meltano, teams can pride themselves not on how complex their solutions are, but how simple and elegant they are. A “great” solution is a solution that is elegant, repeatable, and easily understood by new team members. Everyone can see the work contributed by everyone else, and that code is human readable by design.

Meltano’s DataOps approach makes the hard stuff easy

Want to add Salesforce as a new source? Easy:

✅ Type into a terminal: meltano add extractor tap-salesforce

Want to do the same thing with a friendly UI? Easy:

✅ Go to the meltano web UI, click the option to add a new extractor, and select “Salesforce”.

Want to give everyone in the company the ability to contribute? Easy:

✅ Make the code repository public to all employees – especially the company’s product managers and analysts!

…while no single person has access to break anything? Easy:

✅ Require code review on all changes.

Want to make sure it’s perfect before you deploy? Easy:

✅ Turn on CI/CD with your source provider. Get a full or mini copy of the DB every time you make a change!

Want to undo your mistake? Easy:

✅ Instantly roll back to any point in your history using git.

Like magic! The right tools, fit together just right.

The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they’re indistinguishable from it.

Mark Weiser

Everything should be made as simple as possible, but no simpler.

Albert Einstein

What’s unique with Meltano is that rather than build a more complex platform or a more complex set of tools, we’re making a simpler platform and a more cohesive set of specialized tools. We solve the core data problems in a fundamentally more elegant way. For instance, rather than build a platform with rollback and schema-diff capability, we get out of the way and let source control and CI/CD tools do their job. In future, you won’t be thinking about Singer taps and targets – you’ll just be thinking about source data and where we want it to land. You won’t be thinking about dbt or SQL – you’ll just be thinking about whether your model runs smoothly and meets the business requirements.

The sign of having a great tool is that the tool gets the job done seamlessly and doesn’t distract or get in the way, providing safety and agility at the same time. While I’m not going to claim that Meltano is perfect (not yet! 😉), I do believe Meltano is on its way to being the perfect DataOps platform for any organization – whether you are a junior analyst, a senior data engineer, or a one-person startup.

Let’s build Meltano, together!

I’ve told you now why I’m excited about building Meltano, but the most exciting part is that I get to build it alongside all of you! Whether you are reporting bugs or fixing them, whether you are brand new in your career or have 20 years under your belt: we’re all in this together and we all can benefit from making Meltano the most amazing product it can be!

I do believe we’ll be successful in making DataOps available to the world and we’ll do it with Meltano. 🚀

Let’s do this!

Aaron (“AJ”) Steers

Demo Day Recap: 2021-03-26

Our first Meltano Demo Day of 2021 was a rousing success and everyone had a great time! If you missed it, the recording of our session is available on the Meltano YouTube channel.

Taylor briefly showed off some updates to the main website including our new Community Code of Conduct and our Product Roadmap. He also shared some data about how active the Meltano Slack group is thanks to all of the wonderful people we have in the community.

AJ gave us an excellent walk-through of the Singer SDK and how to get started building your own tap with the cookiecutter template. Checkout this issue for a list of working samples using the SDK. He also highlighted a VSCode extension he uses called TODO Tree to make it easy to find TODOs scattered throughout your codebase.

Then Julian talked us through his latest contribution in v1.71.0: the ability to include arbitrary Python executables within a Meltano project using the utility plugin type. This led to some good discussion about tools like Yoyo and SQLFluff as well as the future of how to run more complicated data pipelines using Meltano. This led to even more discussion about the future of the commands flag to simplify workflows. Checkout issue #2560 to follow along as that develops.

Lastly, Douwe shared the tap, target, and pipeline he built to solve a personal data problem. Using the Singer SDK, he built a tap to pull data from investing.com. He then built a new target for pushing data into Lunch Money (a tool for managing personal finances). Finally he wired it up in a separate Meltano project and is using GitLab CI to schedule a daily sync of tap-investing to target-lunch-money. It was great to see how Meltano can be used to solve personal data integration challenges and gives a preview of how Meltano may be used to solve for the “Reverse ETL” or “Data Pump” use cases.

Overall, we’re quite happy with how Demo Day went and we look forward to hosting many more of these. We’re always listening to feedback and looking for ways to improve, so please contribute to any issue within the main project or chat with us on Slack if that’s more your style. Thank you to everyone who joined the livestream!

Join the conversation!

Our next Meltano Demo Day is on April 9 and we have our next office hours session on Wednesday, March 31. You can always find a list of all our events in the #events channel on Slack.

Office Hours Recap: 2021-03-24

This week in Meltano Office Hours, we heard directly from community members about their experiences running Meltano on Windows, WSL, and Docker. If you missed it, the recording of our session is streaming now on the Meltano YouTube channel.

Our biggest takeaway from the discussion for Windows users is that WSL2 will generally provide you the best and most reliable experience for running Meltano on Windows. That includes being able to run the Web UI locally without requiring Docker. While WSL1 might also work, we don’t recommend it. Just skip directly to WSL2 if you are able to… And then go brag to your Windows and Mac coworkers that your laptop can run both Windows and Linux!

At Meltano, we are always listening to feedback and looking for ways to improve. Based on feedback from the discussion this week, we are tracking the following issues in our issue tracker. If you are interested in contributing to any of these, or if you just want to join the conversation, feel free to drop a comment at the links below or chat with us on Slack if that’s more your style.

(These links are also in the YouTube stream‘s description, along with topic timestamps for quick scanning.)

A big thanks to everyone who joined. And a special thanks to Derek Visch from Auto IDM who served as our resident community expert for Windows and WSL!

Join the conversation!

Our our next office hours session will be on Wednesday, March 31, and we have our Meltano Demo Day tomorrow, March 26. You can always find a list of all our events in the #events channel on Slack.

Hope to see you soon!

AJ Steers

Now Available: Meltano v1.71.0

Today, we are excited to release Meltano version 1.71.0, which:

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.70.0 on February 23:


  • #2544 Add support for utility plugin type
  • #2614 Add mashey variant of tap-zoom


  • #2581 Only expand $ALL_CAPS env vars in meltano.yml config values to prevent false positive matches in passwords

Building Meltano in Public: Bimonthly recap

Last week, it was once again our turn to host a GitLab Group Conversation (a publicly live streamed Q&A on the GitLab Unfiltered YouTube channel) on Meltano!

We used the opportunity to share a recap of:

If you’re curious, check out the presentation on Google Slides and the Q&A on YouTube. The presentation content is also reproduced below, as is an embedded video of the Q&A!

Group Conversation Presentation

2 new team members just joined!

  1. Taylor Murphy joined on March 1
  1. AJ Steers joined on March 8
    • AJ comes from Slalom, where he was a Solution Principal building next-gen products and reusable frameworks to solve today’s hardest data problems
    • As a contributor, he initiated the Singer SDK project: https://gitlab.com/meltano/singer-sdk
    • Announcement blog post: WIP

With Douwe Maan, that makes 3 full-time Meltano team members!

5 releases since the last GC (2021-01-14)

  1. V1.66.0 prevents pipelines from getting stuck in the “running” state forever when their meltano elt process is killed unceremoniously by the operating system or some other mechanism.
  1. V1.67.0 fixes two bugs with meltano schedule run <name>: if the schedule’s meltano elt command fails with a nonzero exit code, it now does as well, and it no longer requires the meltano executable to be in the PATH.
  1. V1.68.0 adds support for entity/attribute selection to tap-gitlab and bumps Airflow to version 1.10.14.
  1. V1.69.0 adds out-of-the-box support for the Quickbooks source and adds support for Airflow 2.
  1. V1.70.0 prevents two pipelines with the same Job ID from running at the same time, adds out-of-the-box support for the Chargebee and Sage Intacct sources, and fixes a bug that prevented Meltano Analyze from working with custom plugins.

17 recent contributions by 9 community members


  1. target-snowflake: Dynamic Precision Fix by Bryan Wise (Halosight)
  2. tap-gitlab:  Implement discovery mode and stream selection by Charles Julian Knight (FIXD)
  3. Add support for Airflow 2.0″ by Michel Radosavljevic (Insurello)
  4. Add `hotgluexyz` variant of `tap-quickbooks` by Hassan Syyid (Hotglue)
  5. Document that tap-postgres depends on libpq by Derek Visch (Auto IDM)
  6. Docs Development Steps NPM run serve:docs Missing Script by Derek Visch (Auto IDM)
  7. Add `hotgluexyz` variant of `tap-chargebee` by Hassan Syyid (Hotglue)
  8. Fix Meltano Analyze with custom extractor by Al Whatmough
  9. Add `hotgluexyz` variant of `tap-intacct` by Hassan Syyid (Hotglue)
  10.  Refactor Job sync implementation by Charles Julian Knight (FIXD)
  11. files-airflow:  Add pipeline details into tags by Pankaj Saini (Let’s Do This)

In development

  1. Support for a generic plugin type for arbitrary python cli plugins by Charles Julian Knight (FIXD)
  2. Add support for shortcut commands for plugins by Charles Julian Knight (FIXD)
  3. Adding pipelinewise target-redshift by Ingo Klose (b.telligent)
  4. Disallow parallel pipelines by default by Charles Julian Knight (FIXD)
  5. Add tap-zendesk variant by Benjamin Maquet (Twilio)
  6. singer-sdk: Fix reading catalog from JSON file by Edgar R. Mondragón (SpotOn)

Other exciting recent and ongoing developments

Recent and projected weekly Slack activity

Join 850+ on Slack!

This week’s priorities

Milestone issue board

Upcoming priorities

Milestones issue board


Group Conversation Q&A

Joining the team: Taylor Murphy shares why he’s excited

Today is my first official day working on the Meltano team! In early February 2021 I announced on Twitter that I would be leaving the GitLab Data Team, where I’ve worked for the past 3 years, and joining Meltano. There was a short thread that accompanied the announcement which I promised to expand upon. I want to dive deeper on a few points in that thread and share more why I’m excited about the project and role.

In May of 2020 Douwe shared the new focus of Meltano: make the power of data integration available to all by turning Meltano into a true open source alternative to existing proprietary hosted ELT solutions. He directly cited my comment from 2018 where I indicated this was a potential avenue Meltano could take to build a real community. (Let’s ignore the part where I said we should create our own Meltano spec! I’m now fully aligned with our embrace of the Singer Spec.)

This renewed focus on being an excellent way to run open source extractors and loaders is part of why I’m so excited. Much of what makes communities like dbt and Airflow great is that they are open source tools that people are using to get real work done. For dbt in particular, the ability to network within the community, achieve real results, and contribute back to the project are part of why it has seen such success.

Meltano can be an excellent tool within the data space for many years to come and I want to share why I believe that.

Open Source, Always

GitLab is an open core company that has a free open source (FOSS) version of its product and is committed to having it be free forever (see the stewardship section of the handbook for more information). This model has served it well and is something we will emulate with Meltano. As Douwe stated in the part 2 of the pivot blog posts: “there will always be a Community Edition, and data integration will forever be a commodity rather than pay to play.” This is a critical aspect of our mission and strategy which makes it an exciting place to work.

I also believe that open source is essential to actually achieving our vision since one company can’t possibly manage and maintain the entire world of extractor and loader combinations. This reality leads to the next reason why I’m excited about the future of Meltano.

Facing the Challenge

Meltano is an exciting project too because it’s tackling a hard problem. As I said in a tweet a while ago, much of data engineering is digital plumbing – it’s critical work that often goes unnoticed and underappreciated until something breaks. It seems like it shouldn’t be as difficult as it is, but there is a long tail of data sources and data problems that are all slightly unique. 

Solving hard problems that not a lot of people want to touch has been a through-line in my career and it’s something that excites me. The process can be arduous but the feeling of accomplishment from crafting great solutions to challenging problems that help real people is immensely fulfilling. And as Meltano grows I think the scope of data problems it solves can continue to grow as well (keep in mind Meltano does a lot more out of the box than just extraction and loading even if we aren’t focused on that from a development perspective right now!).

But if we had to solve these problems alone then I don’t think I would be as excited. We can’t meet the needs of everyone if everything is closed source and internal only. Open source is part of the solution to the challenge, but so is collaborating with a large network of data professionals.

Building a Community

The data community has been wonderful for me personally and professionally. I’ve met and connected with so many wonderful people via the dbt and Locally Optimistic communities (and Twitter!) that I never would have known had I just focused on my local Nashville community. It’s also enabled me to share my work from the GitLab Data Team in a transparent way that has helped numerous people in the community. There’s no better feeling than being able to answer someone’s question in Slack with the answer “here’s how we did it and here’s a link to our code – let me know if you have more questions!”. 

I want Meltano to build on these communities with a focus on open source data integration. We see the project as working very well with dbt and other open source tools and I know it can be a great place for data engineers and other data professionals to collaborate on solving real problems. And it will help address that long (infinite) tail of data integration challenges by crowdsourcing the efforts via open source software.

I also share Douwe’s belief that Meltano can make world-class data engineering more inclusive of a broader audience. The “standard” data stack of Fivetran, Airflow, Snowflake, Census, and Looker can be expensive and much of it is pay to play. Much like how GitLab enables people to have a world-class DevOps experience with a free product, Meltano should do the same for DataOps.

Elevation of the Data Profession

I recently gave a talk with the inimitable Emilie Schario at Coalesce 2020. The gist of it was that Data Teams should view themselves as building a Data Product that serves their entire organization. This view is born out of the deep belief that most organizations aren’t realizing the full value of their data and that to close that gap a rethinking of how teams work is needed. 

I see Meltano as being able to help enable this vision. The data profession and lifecycle tool chain is still in its early days with companies and startups everywhere getting early funding (dbt, Census, Airbyte, Hightouch, Fivetran, etc.). Many of these companies are addressing real pain points in the modern tool chain and will probably be successful businesses. But a proliferation of closed-source point solutions won’t enable the next level of data awareness that is required to close the data utility gap. None of these tools have the community or broad vision that I believe Meltano can and will have.

I want Meltano to follow the GitLab model: be truly excellent at a core piece of technology and then broaden to include more of the workflow in a way that makes sense for the community (and eventually customers). Meltano will be excellent at running Singer extractors and loaders while also enabling people to easily build their own (see our SDK effort). This will be the core of what Meltano does well.

The next level though is having a meta understanding of how data is flowing throughout your entire Data Product. It’s great that you can connect Fivetran to Snowflake and Snowflake to Census (this blog post is a compelling discussion on how they close the loop). But there is a not-so-hidden challenge in that diagram: it is difficult to have a holistic understanding of what’s happening throughout the stack. A single, metadata-focused view of how data is flowing requires a lot of data engineering work not pictured. While Meltano doesn’t solve this now, we’re building the tool in such a way that we can address this challenge better without requiring you to check multiple tools or manage the metadata yourself.

While good tools don’t solve all of your organizational challenges with data, I believe great tools can go a long way towards improving the current state of data operations. 

Looking to the Future

I believe the next 10 years of the data profession has so much to offer and I want to be a part of building the tooling for that journey. Meltano can be a central player in the data community, helping people build their careers and get real work done. GitLab has shown the power of building a true community and enabling collaboration on a scale never before seen. It’s also built an amazing business that enables it to support the open source project and drive digital transformation in thousands of organizations. 

Meltano will do the same, but first we want to focus on building a strong community. A strong community means:

  • It’s easy to get up and running with Meltano to get real work done
  • There are real people working on the project who are kind and helpful (Douwe, myself, and 1 more soon!)
  • There are tools that exist to make it easy to contribute to the project and to build new taps and targets (see our Singer SDK effort)

We’re making great progress on these fronts and will continue to expand our efforts as the community grows. 

There are also some additional questions that I’m excited to be thinking about too:

  • How do we enable the metadata-first view on Meltano so that data about your data flow is easy to use? (See this issue if you have thoughts!)
  • How can we help build trust in community taps and targets with an open testing and validation framework, with the goal of having a central place to learn about the behavior, supported features, and maintenance status of all taps and targets in the ecosystem? 
    • Airbyte’s connector health page is an example of what I’d like to see, but we’d need to implement it in a way that’s open to the community. See this issue to add your thoughts!
  • How do we build Meltano as a business in a way that’s a win-win for everyone?

And more! I don’t have the full answers and we need the community to help us answer these questions. I’m excited to continue bringing the GitLab values to the project and build upon the great foundation that Douwe and many others have started. 

Questions or feedback? Come say hi to me on our Slack or on Twitter, or open an issue in the Meltano project.