Meltano

Office Hours Recap: 2021-04-14

In office hours this week, we discussed our vision for the upcoming SingerHub as well as well as several SDK topics such as parent-child streams. Keep scrolling for the recap, or catch up on what you missed using the recording below.

Click through to YouTube for detailed timestamps on each topic in the video’s description.

Envisioning a Singer “Hub” for plugin discovery

The most exciting topic for me this week was a resounding confirmation of the many ways SingerHub can improve the process of discovering taps and targets within our ever-growing Singer community. Above and beyond what we have in github.com and singer.io today, we discussed strategic metrics and metadata which SingerHub could expose to users upfront, greatly streamlining the plugin discovery and adoption process. For example, SingerHub could help users evaluate taps based on metrics such as download numbers from PyPi, recency of commits on GitHub, anonymous usage metrics from Meltano, and many others. In addition, SingerHub can generate automated documentation for catalog and config data, which otherwise would have to be discovered by manually scanning GitHub documentation or (gasp!) the plugin code itself. We heard this week a resounding confirmation that community members share our desire to streamline this process and bring rich metadata and metrics together all in one place.

I won’t try to recap everything we discussed here, but please join the discussion using our new #singerhub channel in Slack, and stay tuned for more announcements from Taylor Murphy on SingerHub and how it fits with the Meltano open source vision for DataOps!

New SDK feature proposals

In addition to the Singer Hub vision work mentioned above, we also discussed several SDK related features. Those included a “demo mode” in the SDK, PII tracking, advanced parent-child relationships, and a new idea for explicitly declaring unsupported streams. If your are interested in learning more or contributing to these features, we highly encourage you to join the discussion, either using the links above or by posting your thoughts to our dedicated #sdk channel in Slack.

We hold Office Hours every Wednesday at 9AM Pacific, 16:00 UTC.

Hope to see you soon!

AJ

Demo Day Recap: 2021-04-09

The first Demo Day of April was a jam-packed and an absolute blast! If you missed it, the recording of the livestream is available on the Meltano YouTube channel.

Taylor briefly shared some of the recent documentation updates and blog posts including the SDK Launch, our interpretation of the Singer Spec, and our vision for the future of Meltano.

AJ then shared improvements to the SDK including the release of v0.1.1, the cookiecutter template, and the dev guide.

Ken Payne from Tails.com shared how he’s used the Singer SDK to build a tap to pull data from Tableau. This is a great example of how the SDK simplifies the code required to build a high quality tap.

We also had Tulkin from Applied Labs talk about their announcement of Singerly, which is a hosted platform for Meltano and Airflow. That led to some discussion on the details of Singerly including environment variables and metrics options. There was also a discussion of the upcoming SingerHub.

Lastly, Ken shared a bit about how they deploy Meltano. We plan to have him back to dive in deeper on this topic.

Overall, this was a great Demo Day and we look forward to hosting many more of these. We’re always listening to feedback and looking for ways to improve, so please contribute to any issue within the main project or chat with us on Slack if that’s more your style. Thank you to everyone who joined the livestream!

Join the conversation!

Our next Meltano Demo Day is on April 23 and we have our next office hours session on Wednesday, April 14. We hope to see you there!

Our Vision for The Future of Meltano

Recently, the Core Team met to discuss our visions of what Meltano could become over the coming years. It was a wonderful, multi-hour discussion and all of us left feeling excited about the potential and eager to manifest that future. In this post we wanted to share the highlights of what we talked about and how we think about the future of Meltano.

Constant Commitment to Open Source and the Data Community

While we have a lot of features on the roadmap, one of the things we talked about is the importance of our values and the community. To that end, we recognized that having a healthy open source ecosystem and a strong data community are going to be key drivers of our success over the long term. 

One part of our mission with Meltano is to level-up the Data Profession. This means recognizing the value and competitive advantage that Data Professionals and Data Teams bring to an organization. Many Data Teams were built as an afterthought when an organization realized it “needed to do something with its data”. The state of tooling for Data Teams reflects that reality: it is disjointed, challenging to use, and squanders a huge opportunity for organizations everywhere.

We believe that building open-source software and fostering and supporting OSS within the data ecosystem is essential to achieving our full mission. OSS has elevated the best practices within software development and we know it can have the same effect on the Data Profession. We believe Singer will be the de facto standard for any data integration challenge. We also believe that Open Source will win many parts of the data stack. We want to integrate with and support those tools as well.

We also want the Meltano community to support all data professionals, not just Data Engineers. We aim to provide training and education for the entire data lifecycle, including for Meltano specifically. As the data profession grows into fully embracing DataOps we want to help people onboard to this new and better way of working. With so many open source tools available it can be daunting to know where or how to get started. We aim to make the tooling as understandable as possible while giving people the confidence to jump in and get working with a thoughtful, curated stack that supports modern best practices.

We recognize the fantastic responsibility of stewarding an open source project and community. We aim to do what’s best for Meltano and the larger data community so that we can sustainably build a great tool for years to come.

In 5 years – Be the One-Stop Shop for Data Teams

In the long-term we aim to go big. We believe that Meltano can become the one-stop shop for the majority of data projects. Here are some of the key things we want to see in Meltano less than 5 years from now.

Deep First Principles Integration – Issue #2667

We believe it will be critical to have DevOps best practices in the DNA of the product. This means having a strong notion of version control, CI/CD, and the development experience. It should be easy to have disposable environments for the entire stack (think about Snowflake zero copy clones). We believe these best practices to be important because DevOps tooling enables software teams to move faster with more confidence, reliability, and security – Data Teams will need to move faster as well since high-performing Data Teams are becoming more critical to the success of all businesses.

Fantastic User Interface – Epic #78

While we are building Meltano CLI-first, we know it will be essential to have a great UI for the tool as well. We believe that building for the command line first and growing from there will lead to a better experience and growth path for the product. But for us to be able to compete and bring in a whole new generation of data professionals we will need an excellent user experience with a well designed UI.

Enterprise Ready – Issue #2667

We want Meltano to be a long-lived project with a sustainable business supporting it. To do that we need to sell to large enterprises. This means thinking about features needed by these organizations before we’re ready to sell to them. We believe we can do this in a way that is supportive of the larger data community by following a buyer-based open core business model where only those features that are more relevant for managers and up are what we charge for. We will also want to have a SaaS and a Self-Hosted option that are equivalent to each other.

Go Big!

Meltano has a big future ahead of it and we’re confident in our ability to manifest it. But how do we get there? One iteration at a time. To that end, we also talked about where we want to be within the next year.

Meltano by 2022-H1

By next year we want to have all of the following done (and more, of course!).

Modernized and Enhanced Singer Ecosystem – Epics #87, #85, #80, & #88

Our work with the Singer Software Development Kit (SDK) is well underway and it will become the de facto way to build new taps and targets. Along with the SDK, we will soon be building the SingerHub which will be the easiest way to find working and up-to-date taps and targets. 

These investments into the Singer community will continue with support and enhancement of the protocol itself. We’ve already released a first iteration of our interpretation of the Singer spec. We also plan to update and support optional extensions such as `ACTIVATE_VERSION` and `BATCH` (aka Fast Sync) message types. We’re also thinking about how to have a generic data plugin architecture for Singer so that data can be obfuscated or dropped on the fly.

Enhance the Command Line and Configuration Experience

Because we’re building Meltano with a CLI-first focus, we want to make working on the command line within your Meltano project even better. Bringing better decentralization and composability to a project is a part of that goal. This means making it easier to simplify configuration and have it spread out across your Meltano project by being able to split up your `meltano.yml` file and even import other projects into one another. Since a key part of the promise of Meltano is that it will integrate with many tools, being able to integrate with itself will be a core concept. Making Meltano git-aware is something we’re eager to add as well since it will help with the development and deployment of Meltano.

Metadata, Monitoring, Observability – Epic #93

Meltano as a product sits in a unique position because it is at the start of the lifecyle for any data project. Part of the promise of an all-in-one tool is that it makes it possible to easily understand that status of data anywhere along the value chain. We want to make metadata a native part of the product and build this capability early.

Great Deployment Experience – Epic #28

Deploying Meltano is a critical part of making it usable. We want to make it as easy as possible to deploy Meltano in any environment. To do this we’re going to enhance our dogfooding of the product as well as simplify the deployment process as much as possible. Great documentation will be a key piece of this too.

Full Open Source End-to-End Solution

Out of the box we want to have everything you’d need to get a modern data stack up and running. This means everything from data integration to orchestration, warehousing, transformation, operational analytics (Reverse ETL, if you like), and reporting. We’re going to do this by integrating with tools like Dagster, Prefect, Superset, and Jupyter Notebooks, but also by having support for a modern analytic datastore such as Spark, Clickhouse, or DuckDB. We’re also going to dramatically improve our dbt integration by doing things like autogenerating sources.yml files and adding native support for testing. Having this end-to-end stack in one place will enable unprecedented coordination between the different tools meaning we’ll be in a great place to achieve our long term vision.

The Future is Bright

As you can see, we’re all extremely excited about this space. We want to build a tool for Data Professionals so that organizations can manage and access their data more effectively so they can enable better decisions. We want to level-up the status of the Data Profession so that those who work in it are respected for the true value they can bring. We believe that the data community is growing more technical while the value of data within organizations is becoming more valuable. 

We hope to support the community through the Meltano product but also via the open source ecosystem too. Meltano will be driven by what the community wants and needs, so we need your active involvement! Please collaborate with us on issues, in Slack, and on Twitter for how we can make the product and the community better! 

Office Hours Recap: 2021-04-07

This week in Meltano office hours we discussed how the newly released Singer SDK can dramatically speed up the development of new Singer Taps, as well as our newly launched Singer Spec documentation, which make it easier than ever to understand exactly how the open source Singer framework actually works behind the scenes.

You can watch the full livestreamed session below or on YouTube. Check the annotations in the video description if you want to jump straight to a specific topic.

Meltano Office Hours – 2021-04-07

We hold office hours every Wednesday, 9AM Pacific / 16:00 UTC. For more info, including the link to each week’s Zoom, check join the #office-hours channel in Slack. If you are interested in showing off your latest work, or seeing what others have built, please also join us this Friday for Demo Day!

Hope to see you soon!

AJ

A Simplified Take on the Singer Spec

The Meltano team is excited to share a recent update we made to our documentation of the Singer specification: we’ve created an interpretation of the spec that is easier to understand and includes sections for different audiences. 

Part of our mission with Meltano is to embrace the Singer community by building tools and resources that lift-up the entire ecosystem. Our launch of the Tap SDK on Monday is one example of that effort. 

Our Singer Spec interpretation is the next example. One of the challenges we saw for new people to the Singer ecosystem is that it can be difficult to “grok” everything that’s going on with the spec as it’s written on on the singer-io GitHub group. We identified this as one of the challenges slowing down the community and took it upon ourselves to write a version that is more focused on the core of the spec and is free of heavily-intertwined, vendor-specific information.

To be clear, we have no intention of replacing the Singer spec. This interpretation is meant to consolidate some of aspects of the spec that were spread across multiple pages and bring them together into a single resource for ease of use. 

Our intention going forward is to move this out of the Meltano website and onto the SingerHub, once it’s launched, so that it’s even clearer that the spec isn’t tied to any specific company or project. 

We hope you find it valuable! If you have any suggestions please submit an issue or merge request, or join us on Slack to chat more about Singer, Meltano, and open source data integration.

Special thank you Derek Visch and Charles Julian Knight who contributed feedback to an early draft of the document.

Meltano Launches v0.1.0 of the Singer Tap SDK

Today the Meltano team is excited to announce a milestone for the Singer community: the v0.1.0 launch of our Singer Tap Software Development Kit! The SDK is a framework that makes it easier than ever to build high quality data extractors, aka taps. With the SDK, tap developers can take full advantage of the Singer spec without being an expert on it, while enabling them to focus on the code unique to the API or database they are extracting data from.

This is one of the many things we’re working on in the coming months to lift up the entire Singer and data integration ecosystem. In addition to continued development of easy-to-use SDKs, we’re also working to simplify the spec so it’s easier to understand for people new to the community. We’re also building a SingerHub so developers can easily find, use, and contribute to high quality taps and targets. There’s also been huge interest from the community around extending the Singer protocol and we expect our efforts with the SDK and within the ecosystem to make it easy for the spec to grow and evolve.

Genesis of the SDK

As you can see, we’re bullish on the potential of Singer and its community, but we recognized the real challenges of building and maintaining high quality taps and targets. We saw that standardization across taps could be dramatically improved – some would implement all parts of the Singer spec while others did the bare minimum. We also saw how challenging it was to build a new tap with all of the great features of the protocol. 

AJ Steers, who joined the Meltano team last month, saw these problems too and started work on the SDK as a community member. As a consultant, AJ has experienced the challenge of building taps that implemented all parts of the Singer spec without any sort of scaffolding to help him. “Optional” features such as incremental replication and stream/property selection were difficult to add.

The SDK solves all of these problems. Developers now don’t have to become an expert in the spec to write a high quality tap. They can focus on the code that’s unique to the API or database and will get all of the good stuff for free. 

Singer SDK In the Wild

Our goal with the SDK is to decrease the amount of code developers have to write by over 70%. To that end, we have a cookiecutter template which will ask you a few questions on the command line:

> cookiecutter https://gitlab.com/meltano/singer-sdk --directory="cookiecutter/tap-template"
source_name [MySourceName]: mytap
tap_id [tap-mytap]: 
library_name [mytap]:
Select stream_type:
1 - REST
2 - GraphQL
3 - Other
Choose from 1, 2, 3 [1]: 
Select auth_method:
1 - Simple
2 - OAuth2
3 - JWT
4 - Custom or N/A
Choose from 1, 2, 3, 4 [1]:

And then within the code we’ve marked multiple #TODOs to highlight where you need to add the custom code for your source. It’s that easy!

Even before this official release, we had many community members interested in the project who started to use it before it was “ready”.

Edgar Ramirez from SpotOn is developing a tap for the Confluence Content API. Edgar said even before the v0.1.0 release,  “I’m focusing on extracting data from REST APIs at the moment and it’s amazing how quickly you can get your tap to output Singer spec messages. [It’s just] a few lines of code, 90% of which deal with idiosyncratic pagination and authentication, the rest are just stream (name, schema, primary keys) and tap (config schema) declarations”.

Derek Visch from AutoIDM needed a quick proof of concept to show he could make a connection work between BambooHR and ActiveDirectory. “Using the SDK, I had BambooHR data coming through in 2 hours, and that’s with no Python experience.”

John Timeus from Slalom developed a tap for PowerBI Metadata and even our own Douwe Maan shared in our first Demo Day the tap he built to pull data from investing.com.

But don’t just take their word for it – check out our dev guide to get starting making your own tap!

The Future of the SDK

v0.1.0 of the SDK is just the beginning. While it supports the basics you’d expect such as data sync, catalog discovery, and bookmark tracking, we have a plethora of features we want to add to the SDK. This includes supporting stream/property selection, database type streams, auto-generation of documentation, and handling unsorted streams. We aim to have the SDK support everything the Singer spec currently allows for, but we also want to build support for extensions on the spec including the `ACTIVATE_VERSION` and `BATCH` message types. You can see these issues and more in the issue tracker. 

We’re also planning to expand the SDK to include Targets as well as for plugins that can ingest and export Singer valid data. This latter use case enables on-the-fly stream transformations like hashing or aggregation.

We hope you can spend some time building taps with the SDK! Our expectation is that all existing taps will eventually be ported over to the SDK. When this happens, the community will benefit from taps that are easier to maintain, are more consistent across different sources, are under a more permissive open source license, and can immediately benefit from enhancements to the SDK and Singer spec.

What’s Next

Keep an eye out later this week for two more blog posts: the next one will be about the updates we’re making to the Singer spec to make it more understandable and the other will be about our long-term vision for Meltano.

On Wednesday April 7th we’ll have our weekly Office Hours which will be focused on the SDK and on April 9th we’ll have our Demo Day where community members and the core team share what they’ve been working on. In the meantime, join us on Slack to chat more about the project, or file an issue if you have feature ideas or run into any problems. 

Happy building!

Meltano is Special: AJ Steers explains why he’s all-in on Meltano

Earlier this month, I joined the Meltano team full time with two ambitious goals: (1) redefine how organizations think about their approaches to data projects and (2) make this next-gen approach to data freely available to everyone, regardless of budget or the size of the data. From hobbyists working on pet projects, to multi-billion dollar corporations: I believe everyone can benefit from Meltano’s approach.

This post will explain why I decided to join Meltano, why I think DataOps is critical to a team’s success, and what I think is so unique about Meltano in particular.

ETL is dead; Long live EL and T!

To start off, one of the biggest reasons I joined Meltano is because I think transitioning to the new DataOps approach is exciting, and I want to do my part to make transitioning to DataOps a positive, joyful experience. If your data team has not taken the plunge yet, the plunge is coming and you should be getting ready! If you work in data, it is likely some time within the next 12 months, someone is going to excitedly come ask you about DevOps, it’s younger cousin “DataOps”, and CI/CD: they’re going to ask you if your data team has taken that plunge into the world of DataOps or if you are still using the old-style traditional ETL tools. I’m here to tell you life is much nicer here on the other side.

Addressing two important questions about the future of data teams

You might be in the data space yourself and you might be skeptical of the direction that tools like Meltano and dbt are moving in. Or maybe DevOps, git, and CI/CD seem like too much to learn. I want to address two of these concerns of these before I go further.

Objection #1: My data team aren’t coders. Do we really need git and CI/CD?

First of all, I know this is going to sound controversial, but your data team are already coders. They just might not yet have the tools they need to support them writing good code. In our own way, we are all coders whether we write code directly or use a tool to write the code for us.

As long as two or more people are working together towards a common goal, there will always be merge conflicts to resolve and code reviews to perform. Have you ever emailed a coworker a copy of a word document, only for both of you to make changes and then stumble through how to merge those changes back again into a single copy? That’s called a “merge conflict” – resolving it made you a coder! Have you enabled “track changes” on a document so someone else could review those changes one at a time? That was a code review and in that moment, you were a coder! These are exactly the types of source control problems which benefit from git and CI/CD. Historically, however, very few of us in the data community have had access to tools that managed our code effectively. That is, until now.

For anyone still nervous about git or “coding”, please consider checking out the Git and GitHub for Poets series. It’s clever and entertaining and does a great job demystifying all the DevOps technical jargon.

Objection #2: But isn’t coding hard? What about the move towards “no-code” solutions?

I want to address this point because I hear this question often in the data space. While we all want to be writing less code, more elegant code, and more efficient code, we can’t get away from code altogether; we can only invent simpler and more accessible languages to deal with it, and then build great tools on top of those languages. The Meltano UI, for instance, literally writes EL code for us. Afterwards, we’re still free to modify the code ourselves if we wish, track changes, and even undo changes.

Rest assured: the languages you learn when building a Meltano project are not going away anytime soon. These are the same languages that I’d bet on my 5 year old son learning in high school ten years from now: SQL, YAML, and Python. These are not only among the most popular languages out there, but they are also among the easiest that one can learn today! (Yes, much easier than French! 😅)

Why I believe Meltano is special

As I think about why Meltano is special, there are three big reasons that come to mind: specialization, composition, and collaboration.

Meltano treats EL and T as distinct problem types (Specialization)

Meltano understands that the challenges with EL (“Extract-and-Load”) and T (“transformation”) are each unique. As such, Meltano handles each with a targeted best-in-class approach for each: the powerful Singer platform as our EL framework and the revolutionary open source tool dbt as our data transformation layer. Meltano itself is the cohesive solution for the entire end-to-end data project. Looking to the future, beyond EL (Singer) and T (dbt), Meltano will similarly integrate with other great tools like Superset for reporting and BI and Great Expectations for validation and governance.

Meltano brings it all together in one place (Composition)

Despite the distinct and unique challenges of each stage of EL and T, Meltano knows that data professionals want a single home for the entire data project, with clear transitions and handoff between each stage of the process. For instance, today Meltano recommends dbt transformation packages based upon the data sources in your project. There is plenty of room for us to continue expanding this functionality in the future.

Meltano is Pro-Team, because… DataOps! (Collaboration)

If you haven’t heard the word “DataOps” before, please allow me to formally introduce you: “DataOps” is simply “DevOps for Data” – and at the same time it is also so much more. The term “DevOps” was coined in 2009 as an agile-driven movement in software development, automating the “ops” part of development and putting source control systems to work automating those steps we don’t need to be spending time on manually. Deploying, packaging, testing, rolling back: all of those tasts are handled automatically by what we now call “CI/CD”. All we need to do is write some tests, use automation to continually retest those tests, and then finally we can trade our manual deployment headaches for a streamlined code review and approval flow. With DataOps, anyone can send a proposed code change, code owners can review that change, and the systems literally do the rest.

With DataOps and Meltano, teams can pride themselves not on how complex their solutions are, but how simple and elegant they are. A “great” solution is a solution that is elegant, repeatable, and easily understood by new team members. Everyone can see the work contributed by everyone else, and that code is human readable by design.

Meltano’s DataOps approach makes the hard stuff easy

Want to add Salesforce as a new source? Easy:

✅ Type into a terminal: meltano add extractor tap-salesforce

Want to do the same thing with a friendly UI? Easy:

✅ Go to the meltano web UI, click the option to add a new extractor, and select “Salesforce”.

Want to give everyone in the company the ability to contribute? Easy:

✅ Make the code repository public to all employees – especially the company’s product managers and analysts!

…while no single person has access to break anything? Easy:

✅ Require code review on all changes.

Want to make sure it’s perfect before you deploy? Easy:

✅ Turn on CI/CD with your source provider. Get a full or mini copy of the DB every time you make a change!

Want to undo your mistake? Easy:

✅ Instantly roll back to any point in your history using git.

Like magic! The right tools, fit together just right.

The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they’re indistinguishable from it.

Mark Weiser

Everything should be made as simple as possible, but no simpler.

Albert Einstein

What’s unique with Meltano is that rather than build a more complex platform or a more complex set of tools, we’re making a simpler platform and a more cohesive set of specialized tools. We solve the core data problems in a fundamentally more elegant way. For instance, rather than build a platform with rollback and schema-diff capability, we get out of the way and let source control and CI/CD tools do their job. In future, you won’t be thinking about Singer taps and targets – you’ll just be thinking about source data and where we want it to land. You won’t be thinking about dbt or SQL – you’ll just be thinking about whether your model runs smoothly and meets the business requirements.

The sign of having a great tool is that the tool gets the job done seamlessly and doesn’t distract or get in the way, providing safety and agility at the same time. While I’m not going to claim that Meltano is perfect (not yet! 😉), I do believe Meltano is on its way to being the perfect DataOps platform for any organization – whether you are a junior analyst, a senior data engineer, or a one-person startup.

Let’s build Meltano, together!

I’ve told you now why I’m excited about building Meltano, but the most exciting part is that I get to build it alongside all of you! Whether you are reporting bugs or fixing them, whether you are brand new in your career or have 20 years under your belt: we’re all in this together and we all can benefit from making Meltano the most amazing product it can be!

I do believe we’ll be successful in making DataOps available to the world and we’ll do it with Meltano. 🚀

Let’s do this!

Aaron (“AJ”) Steers

Demo Day Recap: 2021-03-26

Our first Meltano Demo Day of 2021 was a rousing success and everyone had a great time! If you missed it, the recording of our session is available on the Meltano YouTube channel.

Taylor briefly showed off some updates to the main website including our new Community Code of Conduct and our Product Roadmap. He also shared some data about how active the Meltano Slack group is thanks to all of the wonderful people we have in the community.

AJ gave us an excellent walk-through of the Singer SDK and how to get started building your own tap with the cookiecutter template. Checkout this issue for a list of working samples using the SDK. He also highlighted a VSCode extension he uses called TODO Tree to make it easy to find TODOs scattered throughout your codebase.

Then Julian talked us through his latest contribution in v1.71.0: the ability to include arbitrary Python executables within a Meltano project using the utility plugin type. This led to some good discussion about tools like Yoyo and SQLFluff as well as the future of how to run more complicated data pipelines using Meltano. This led to even more discussion about the future of the commands flag to simplify workflows. Checkout issue #2560 to follow along as that develops.

Lastly, Douwe shared the tap, target, and pipeline he built to solve a personal data problem. Using the Singer SDK, he built a tap to pull data from investing.com. He then built a new target for pushing data into Lunch Money (a tool for managing personal finances). Finally he wired it up in a separate Meltano project and is using GitLab CI to schedule a daily sync of tap-investing to target-lunch-money. It was great to see how Meltano can be used to solve personal data integration challenges and gives a preview of how Meltano may be used to solve for the “Reverse ETL” or “Data Pump” use cases.

Overall, we’re quite happy with how Demo Day went and we look forward to hosting many more of these. We’re always listening to feedback and looking for ways to improve, so please contribute to any issue within the main project or chat with us on Slack if that’s more your style. Thank you to everyone who joined the livestream!

Join the conversation!

Our next Meltano Demo Day is on April 9 and we have our next office hours session on Wednesday, March 31. You can always find a list of all our events in the #events channel on Slack.

Office Hours Recap: 2021-03-24

This week in Meltano Office Hours, we heard directly from community members about their experiences running Meltano on Windows, WSL, and Docker. If you missed it, the recording of our session is streaming now on the Meltano YouTube channel.

Our biggest takeaway from the discussion for Windows users is that WSL2 will generally provide you the best and most reliable experience for running Meltano on Windows. That includes being able to run the Web UI locally without requiring Docker. While WSL1 might also work, we don’t recommend it. Just skip directly to WSL2 if you are able to… And then go brag to your Windows and Mac coworkers that your laptop can run both Windows and Linux!

At Meltano, we are always listening to feedback and looking for ways to improve. Based on feedback from the discussion this week, we are tracking the following issues in our issue tracker. If you are interested in contributing to any of these, or if you just want to join the conversation, feel free to drop a comment at the links below or chat with us on Slack if that’s more your style.

(These links are also in the YouTube stream‘s description, along with topic timestamps for quick scanning.)

A big thanks to everyone who joined. And a special thanks to Derek Visch from Auto IDM who served as our resident community expert for Windows and WSL!

Join the conversation!

Our our next office hours session will be on Wednesday, March 31, and we have our Meltano Demo Day tomorrow, March 26. You can always find a list of all our events in the #events channel on Slack.

Hope to see you soon!

AJ Steers

Now Available: Meltano v1.71.0

Today, we are excited to release Meltano version 1.71.0, which:

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.70.0 on February 23:

New

  • #2544 Add support for utility plugin type
  • #2614 Add mashey variant of tap-zoom

Fixes

  • #2581 Only expand $ALL_CAPS env vars in meltano.yml config values to prevent false positive matches in passwords