This morning, we announced Meltano has reached our v1.0 stable release, and the videos below are great for new and potential contributors looking to see our release process.
Release Process
Meltano’s release process is documented here, and in this video we talk through the steps as well as a tricky issue with version 1.0 (which is actually 1.0.1 in PyPi because of a release back in February that took the 1.0 tag by accident):
Post Release Speedrun
In addition to automated testing, after each release we perform and record a “speedrun” (a term we’ve stolen from expert gamer videos) to make sure the user experience is quick and integrated.
We hope you enjoy v1.0, and that you will check out all the improvements we’ve made to Meltano!
Today we’re happy to announce Meltano’s version incrementation to 1.0, signifying a point of product maturity that offers a solid end-to-end user experience to extract data and turn it into dashboards.
Meltano began as an internal skunkworks project at GitLab ~18 months ago, and over the past 6 months the core team and contributor community have worked together to reach this milestone. We are grateful for all these contributions, and appreciate our colleagues inside GitLab for their warm support for our unconventional “internal startup” approach (more on this in a future post!).
What It Means to Make It To “V1”
Unlike GitLab, which began with a focus on version control and gradually made the transition to managing the entire DevOps lifecycle, the vision for Meltano has been about simplifying the DataOps workflow from day one. With this big vision, we have grappled with the complexity that comes from making choices on behalf of users and needing to satisfy a range of technical know-how levels from experienced data engineer and scientist to freshly minted analyst.
At the beginning of July, we established key criteria for reaching v1.0 to address these challenges and deliver a working product:
UI-based product that is fully usable with the mouse, no command line or coding required (DigitalOcean 1-Click App)
End-to-end experience that takes a user from first data extraction to dashboard visualization in minutes (YouTube speedruns)
Stable release we can guarantee backward compatibility to and reliable support for (logging, upgrade from the UI, extensively dogfooded and documented with all major bug resolved)
Do not invest much more in proprietary Meltano tech for Meltano Model and Meltano Analyze, we should find open source solutions for these phases and integrate them instead
While this might sound like a simple list on its face, the surface area of Meltano as a data pipeline management tool is already large enough that the work required to ensure an end-to-end experience was the bulk of the heavy lifting.
We put the focus on these areas above all else because our core KPI to measure whether Meltano is working is our ability to consistently grow MAUI. With the release of Meltano v1.0 we now offer a product experience worth coming back for and we can’t wait to see what you’ll create!
We are so happy to announce that Meltano is now supported as a one-click installer within the DigitalOcean Marketplace, enabling users to get started with Meltano in minutes with a pre-configured DigitalOcean Droplet in less than 60 seconds.
Install Meltano on DigitalOcean
Ready to take Meltano for a spin? Get started with these simple instructions:
Once you have Meltano up and running, our Getting Started Guide provides a walkthrough of Meltano’s data-to-dashboard capabilities, so you can begin extracting data, building models, and performing analysis in minutes!
We are grateful to our partners at DigitalOcean, Nick Wade, Ryan Quinn and Daniel Levy for helping us get this released. The DigitalOcean Marketplacelaunched earlier this year, and we are thrilled to join companies like GitLab, Docker, CloudBees, WordPress and many more.
If this is your first time exploring Meltano for your company’s data pipeline management, you can follow our Installation Guide and Getting Started Guide to get going in minutes!
New
#950 Removed the Analyze connection configuration: Meltano will now infer connections out of each loader configuration.
#1002 Analyze UI now displays the Topic’s (analysis model’s) description text if applicable
#1032 Add ‘Model’ and ‘Notebook’ to main navigation to communicate that Meltano plans to empower users with modeling and notebooking functionality
#949 Add “Log” button and dedicated sub-UI for tracking an ELT run’s status more granularly
#932 Meltano can now be upgraded from the UI directly.
Changes
#1045 Make it clear that ‘meltano add’ is not hanging while installing plugins
#1000 Update Getting Started guide with updated screenshots and content
#854 Charts now use pretty labels rather than the ID
#1011 Removed “Catch-up Date” in favor of default “Start Date” of extractor
#1002 Update discovery.yml with explicit kind: password metadata (we infer and set input types of password as a safeguard, but the explicit setting is preferred)
#1049 Change default target-sqlite database name to warehouse to not conflict with system database
#949 Update the way Meltano handles logs for ELT runs: Every elt run is logged in .meltano/run/logs/{job_id}/elt_{timestamp}.log. That allows Meltano to keep logs for multiple, or even concurrent, elt runs with the same job_id.
#949 Update “Create Pipeline” redirect logic based on the previous route being ‘transforms’ (this is a UX win setting up the user with the sub-UI for the next logical step vs. requiring a manual “Create” click)
#1051 Automatically set SQLALCHEMY_DATABASE_URI config to system database URI
Fixes
#1004 Fix error when deselecting last attribute in Analyze
#1048 Fix various actions that should have been mutations and did minor code convention cleanup
#1063 Fix the “Explore” button link in Dashboards to properly account for the namespace
Breaks
#1051 Remove MELTANO_BACKEND e.a. in favor of –uri CLI option and MELTANO_DATABASE_URI env var
#1052 Move system database into .meltano directory to indicate it is owned by the app and not supposed to be messed with directly by users
If this is your first time exploring Meltano for your company’s data pipeline management, you can follow our Installation Guide and Getting Started Guide to get going in minutes!
New
#1014 Meltano now logs all output from each meltano elt run in a log file that uses the unique job_id of the run. It can be found in .meltano/run/logs/elt_{job_id}.log.
#955 Establish baseline for demo day and how they should be run
Changes
#891 Contributors can run webapp from root directory
Fixes
#1005 Fix installed plugins endpoints listing identically named plugins of different types under wrong type
If this is your first time exploring Meltano for your company’s data pipeline management, you can follow our Installation Guide and Getting Started Guide to get going in minutes!
Changes
#987 Update routing to match labels (verbs vs. nouns) in effort to subtly reinforce action taking vs. solely “thing” management
If this is your first time exploring Meltano for your company’s data pipeline management, you can follow our Installation Guide and Getting Started Guide to get going in minutes!
New
#976 Route changes will update page title in the web app
