The Journey to Meltano Version 1.0

This post was originally published on Medium

This week we are down to working through the final requirements for announcement Meltano v1.0 and it’s been a long road to get to this point.

I’m so proud of our team for fighting through the mental fog that all too often accompanies a complex and wide-ranging product vision to reach clarity about our order of operations. Sticking to the guidelines below has been key delivering value to users, and I hope readers will check out what we’ve built.

We’re on track to formally release v1.0 in early October, and you can check out the Meltano roadmap for a detailed breakdown of remaining issues.

Usable End-to-End, Without the Command Line

Meltano is an internal startup within GitLab, and our goal is to grow MAUI (monthly active UI users) 10% week-over-week.

When I joined the team in February I quickly discovered something that made hitting this goal very challenging: in our existing user-adoption workflow most command-line (CLI) users were not ever making it to the UI.

In the data world, many analysts receive either a login to a SaaS tool or a link (or laptop) where software has already been installed and configured for them by the IT department or a software engineer. To drive adoption of Meltano as a self-hosted tool for managing data analytics pipelines from data ingestion to dashboard, we knew we’d need to drive bottom-up adoption by people who wouldn’t want to be burdened with these extra steps.

Users who would like to install and/or use Meltano from the command line can still do so, but when v1.0 releases this will no longer be required.

Deployable to the Cloud in a Single Click

Another major barrier we wanted to resolve in service of data analysts adopting Meltano is hosting. While Meltano can be run locally on your laptop/desktop using a virtual environment, as soon as you are looking to pull very large data sets this can become problematic from a performance and/or security perspective.

We are working to offer several one-click installation options on cloud hosting platforms, and our first submission will be to Digital Ocean’s Droplet marketplace. In the process of working on our submission, we’ve also documented the steps required to deploy Meltano on a Digital Ocean Droplet as an advanced user, but look forward to simplifying all of this into a pre-made image you can install with a single click.

Everything You Need Comes Installed

Earlier Meltano users will remember the many steps required to install various pieces of the pipeline. Meltano v1.0 will bundle what we believe are the best-in-class open source software available for each step in the pipeline: data taps and targets from Singer, transforms from DBT, and orchestration from Airflow.

We still have work to do post v1.0 to make Jupyter Notebooks integration even easier (advanced users can check out this tutorial) and we are looking to swap out the Meltano Models and Meltano Analyze steps with open source solutions rather than what we’ve built (email if you have suggestions or are working on a project that might be a fit).

Thank you for following along with our adventure building Meltano!

We are a 6-person startup within GitLab, and your engagement and support helps us learn what is important and keeps us motivated everyday.

You can find us on Twitter @meltanodata and subscribe to our newsletter here

Published by Danielle Morrill

General Manager of Meltano at GitLab