Is this is your first time exploring Meltano? Get started with $50 of free credit on DigitalOcean with our new 1-Click App. Follow the Getting Started Guide to get going in minutes!

New

• #1285 Add link to YouTube guidelines for release instructions
• #1222 Include static application security testing (SAST) in the pipeline
• #1164 Add “transform limitations” message to Transform UI
• #1272 Add Vuepress plugin to generate a sitemap on website build
• meltano-marketing#89 Adds basic title and meta descriptions to all public-facing website & documentation pages.

Changes

• #1239 Update header buttons layout on small viewports
• #1019 Automatically update package.json file versions
• #1253 Do not allow meltano command invocation without any argument
• #1192 Improve helper notes associated with each Extract, Load, and Transform step to better communicate the purpose of each
• #1201 Improved “Auto Advance” messaging regarding Entity Selection. We also doubled the default toast time to improve likelihood of reading feedback.
• #1191 update Google Analytics extractor documentation to explain how to set up the Google Analytics API, and remove duplicate instructions from the Google Analytics API + Postgres tutorial
• #1199 Add example and sample CSV files to the CSV extractor documentation
• #1247 Update the Loading CSV Files to a Postgres Database Tutorial to use Meltano UI for setting up the Extractor and Loader, running the ELT pipeline and analyzing the results. Also provide all the files used in the tutorial (transformations, models, etc) as downloadable files.
• [#1279] Revise “Roadmap” section of the docs with clarified persona, mission, vision, and re-order content
• #1134 Update the GitLab API + Postgres. Include video walk-through and update the end to end flow to only use Meltano UI.
• #95 Update the DigitalOcean CTA to go to the public directory page for the Meltano droplet
• #1270 Main navigation “Pipeline” to “Pipelines” to reinforce multiple vs. singular (conflicts a bit with the verb approach of the other navigation items but we think it’s worth it for now)
• #1240 Provide clarity around how Airflow can be used directly in documentation and UI
• #1263 Document lack of Windows support and suggest WSL, Docker

Fixes

• #1259 Fix meltano elt not properly logging errors happening in the ELT process
• #1183 Fix a race condition causing the meltano.yml to be empty in some occurence
• #1258 Fix format of custom extractor’s capabilities in meltano.yml
• #1215 Fix intercom documentation footer overlap issue.
• #1215 Fix YouTube iframes to be responsive (resolves unwanted side-effect of horizontal scrollbar at mobile/tablet media queries)

Video: 1.5.0 Demo Day

Video: 1.5.0 Speed Run

---

Instructions for upgrading to the most current version of Meltano are available in our documentation.

To see the full history of improvements to Meltano, please review our CHANGELOG

Meltano is a small startup within GitLab, and in order to be successful we have chosen to ruthlessly focus on a serving a single person. We are narrowing the target persona for Meltano to a person we will call The Founder to build a better and more focused product.

Our target persona has the following traits:

• One busy person at a startup using Meltano in single player mode
• They have access to all systems and data across the company
• New to data (does not write code, queries, etc.)
• Need to do analysis to run the business
• Needs to do both engineering tasks and analyst tasks because there is nobody else

What they are not:

• Don’t have “analyst” in their job title. They perform that function because they have to, and likely are CEO/founder running a department that has a lot of SaaS tools and data, likes sales or marketing. They are setting up the foundational systems in the company.
• Don’t have technical know-how or the time for setting up a server, using the command line, writing code, creating custom taps, targets, transforms or models. They are running their company, and have a thousand other things to do.

As a result of this change, we are auditing our product and marketing this week to re-align with their needs. We have already updated the front page of the Meltano website to reflect this change, and will continue to iterate.

How We Got Here

As we continue to build Meltano in public, both as an open source project and also as a highly transparent team who shares our meetings on our YouTube channel and monthly metrics on our blog, I’d love to share a bit more about how we worked together to get things back on track. Maybe you’ll face a similar situation in the future and try some of the strategies that worked for us.

Something’s Not Quite Right

Following the launch of Meltano v1.0, we spent a few weeks on project hygiene: working on bugs, addressing technical debt, and curating our issues to come together on the theme for the next major milestone. As the fog of a major release cleared and we reconnected in conversations about the long-term vision for Meltano, it became clear our individual perspectives on the target persona for the project had diverged and as a result it was causing some decision-making thrash for a small team that is usually pretty in sync.

As anyone who builds things for a living knows, each team member is an inventor in their mind first and often if there is a gap in the stated plan we can begin to imagine a world of possibilities. It’s only human! However, this can quickly lead to major challenges when it comes to what needs to be prioritized. We have our goal to grow MAUI, but that alone is not enough. Having a crisp shared understanding of who those users should be is crucial, because “not all MAUI is created equal”.

Getting Back on Track

Last week, we spent time discussing what each of our perceptions of the target user were and also reconnected with the foundation story of the Meltano project. While this was not a consensus-building activity, talking through the disconnect helped generate a deeper level of understanding, buy-in, and clarity across the team. In a fast moving startup, it can be tempting to skip steps everywhere — but making sure your cofounders and team members feel they’ve been consulted and heard is an investment in the long-term trust and commitment that make it possible to stay motivated to work on hard problems.

I feel extremely fortunate to be part of the Meltano team, and grateful for the maturity and empathy it takes to say, “I don’t think this is clear, let’s pause and get it right”. So often when there is confusion or conflict the strong emotions associated with ambiguity can get in the way of figuring out what’s really going on. The ability to slow down in order to speed up something I’ve come to deeply appreciate in my time working on startups, and this was a great reminder of why that approach is so valuable. Onward!

Is this is your first time exploring Meltano? Get started with $50 of free credit on DigitalOcean with our new 1-Click App. Follow the Getting Started Guide to get going in minutes!

New

#1208 Add description to Plugin definition and updated discovery.yml and UI to consume it

#1195 Add temporary message in configuration communicating their global nature until “Profiles” are implemented

#1245 Add detailed information on the documentation about events tracked by Meltano when Anonymous Usage Data tracking is enabled.

Changes

#1244 Add instructions on how to deactivate a virtual environment

#1126 Minor UI updates to improve clarity around Schedule step and Manual vs Orchestrated runs

#1210 Improved SQLite loader configuration context (name and description)

#1185 Remove majority of unimplemented placeholder UI buttons

#1166 Clarify in documentation that plugin configuration is stored in the .meltano directory, which is in .gitignore.

#1200 Link to new Getting Help documentation section instead of issue tracker where appropriate

#1227 Update Notebook MainNav link to jump to our Jupyter Notebook docs

Fixes

#1075 Fix a bug that caused target-csv to fail.

#1233 Fix the Design page failing to load a Design that has timeframes on the base table

#1187 Updated configuration to support readonly kind to prevent unwanted editing

#1187 Updated configuration to setting resets to prevent unwanted editing

#1187 Updated configuration to conditionally reset certain settings to prevent unwanted editing

#1187 Updated configuration to prevent unwanted editing until we handle this properly with role-based access control

#1187 Updated certain connector configuration settings with a readonly flag to prevent unwanted editing in the UI. This is temporary and will be removed when we handle this properly with role-based access control.

#1198 Fix “More Info.” link in configuration to properly open a new tab via target="_blank"

#1229 Improve extractor schema autodiscovery error messages and don’t attempt autodiscovery when it is known to not be supported, like in the case of tap-gitlab

#1207 Updated all screenshots in Getting Started Guide to reflect the most current UI

---

Video: 1.4.0 Demo Day

Video: 1.4.0 Speedrun

---

Instructions for upgrading to the most current version of Meltano are available in our documentation.

To see the full history of improvements to Meltano, please review our CHANGELOG

As Meltano progresses in maturity, functionality and features and the team continues to grow we are publishing more MRs, updates, and videos than ever before. Starting this month, we will recap our progress here and through the Meltano newsletter.

---

7 Releases in October

In October we released Meltano v1.0, which is thematically focused on delivering an efficient and predictable end-to-end user experience through the self-hosted Meltano UI. Since then, we have continued to iterate and encourage Meltano users to upgrade to the current version to take advantage of the latest improvements.

---

90% Month-over-Month Increase in Meltano Usage

Our goal is to grow Meltano MAUI (monthly active UI users) by 10% week-over-week, which is similar to the growth metric we are challenged to achieve while in Y Combinator. At the target pace (in red) we will have 1,000 MAUI by the end of 2019. While we experienced significant month-over-month growth of MAUI relative to September, we are still behind pace.

As you can see, we’re on track when it comes to command line users but we are lagging behind when it comes to converting them to the self-hosted Meltano user interface and achieving our 10% WoW target.

It is possible that we’re experiencing some degree of measurement error due to analytics opt-outs and some mistakes we made in setting up Google Analytics (which is why you see a big dip in August) that have now been fixed, but that does not change our goal.

To help improve MAUI and get these goals back on track, we added a marketer to the team in mid-October and look forward to seeing the benefits of his work as he continues to ramp up our efforts.

Contributions & Cadence

Thank you to all our contributors, both inside GitLab and across the broader open source community for Meltano. Learn more about how you can get involved by reading our Contributor Guide and join the discussion in the public Meltano Slack community.

MRs Merged by Month

• 

Top Contributors by Opened MR Count

• 

Consolidated CHANGELOG

New

• #1208 Add description to Plugin definition and updated discovery.yml and UI to consume it
• #1195 Add temporary message in configuration communicating their global nature until “Profiles” are implemented
• #1245 Add detailed information on the documentation about events tracked by Meltano when Anonymous Usage Data tracking is enabled.
• #991 Add e2e tests for simple sqlite-carbon workflow
• #1103 Add Intercom to Meltano.com to interact with our users in real-time
• #1130 Add Tutorial for extracting data from Google Analytics and loading the extracted data to Postgres
• #1168 Speedrun video added to home page and new release issue template
• #1182 Add nullable date inputs so optional dates aren’t incorrectly required in validation
• #1169 Meltano now generates the dbt documentation automatically
• #1123 Add first-class “Submit Issue” CTA to help expedite resolution when a running job fails. Also updated the “Log” CTA in the Pipelines UI to reflect a failed state.
• #1121 Add ability to configure listen address of Meltano and Airflow
• #1022 Add “Autorun Query” toggle and persist the user’s choice across sessions
• #1060 Auto advance to Job Log from Pipeline Schedule creation
• #1111 Auto advance to Loader installation step when an extractor lacks entity selection
• #1106 Add description metadata to the GitLab extractor’s Ultimate License configuration setting
• #1057 Auto advance to Entity Selection when an extractor lacks configuration settings
• #51 Update Google Analytics to track appVersion, custom projectId, and to properly use the default clientId. The CLI also now uses client_id to differentiate between a CLI client id (not versioned) and the project id (versioned).
• #1012 Add intelligent autofocus for improved UX in both Extractor and Loader configuration
• #758 Update ‘meltano permissions’ to add –full-refresh command to revoke all privileges prior to granting
• #1113 Update ‘meltano permissions’ to have the ability to find all schemas matching a partial name such as snowplow_*
• #1114 Update ‘meltano permissions’ to include the OPERATE privilege for Snowflake warehouse
• #1020 Update Command Line Tools documentation to reflect a standard format with opportunities for improvement in the future
• #524 There is a new Plugins section on the site to contain all ecosystem related libraries (i.e., extractors, loaders, etc.)
• #51 Add Google Analytics tracking acknowledgment in the UI
• #926 Add step-by-step intructions for using the DigitalOcean one-click installer
• #1076 Enable Log button in pipelines UI after route change or hard refresh if a matching log exists
• #1067 Add Model landing page and update Analyze main navigation to a dropdown displaying the various analysis CTAs associated with each model
• #1080 Add live chat support on Meltano.com website using Intercom.io
• #950 Removed the Analyze connection configuration: Meltano will now infer connections out of each loader configuration.
• #1002 Analyze UI now displays the Topic’s (analysis model’s) description text if applicable
• #1032 Add ‘Model’ and ‘Notebook’ to main navigation to communicate that Meltano plans to empower users with modeling and notebooking functionality
• #949 Add “Log” button and dedicated sub-UI for tracking an ELT run’s status more granularly
• #932 Meltano can now be upgraded from the UI directly.

Changes

• #1244 Add instructions on how to deactivate a virtual environment
• #1126 Minor UI updates to improve clarity around Schedule step and Manual vs Orchestrated runs
• #1210 Improved SQLite loader configuration context (name and description)
• #1185 Remove majority of unimplemented placeholder UI buttons
• #1166 Clarify in documentation that plugin configuration is stored in the .meltano directory, which is in .gitignore.
• #1200 Link to new Getting Help documentation section instead of issue tracker where appropriate
• #1227 Update Notebook MainNav link to jump to our Jupyter Notebook docs
• !1061 Update the Getting Started Guide and the Meltano.com documentation with the new UI and information about job logging and how to find the most recent run log of a pipeline.
• #1213 Add VuePress use and benefits to documentation
• #922 Document the importance of transformations and how to get started
• #1167 Iterate on docs to improve readability and content updates
• #1013 Toast initialization and analytics initialization cleanup
• Compress meltano-logo.png
• #1058 Updated the selected state of grouped buttons to fill vs. stroke. Updated the docs to reflect the reasoning to ensure consistency in Meltano’s UI visual language
• #1068 Replace dogfooding term in docs to speedrun
• #1101 Add new tour video to home page
• #1101 Update design to improve readability and contrast
• #1115 Update ‘meltano permissions’ to not require an identically named role for a given user
• #1087 Fix meltano select not seeding the database when run as the first command.
• #1090 Update the namespace for all plugins. Also the default schema used will go back to including the tap_ prefix to avoid conflicts with existing schemas (e.g. a local gitlab or salesforce schema). This also fixes tap-csv and tap-google-analytics not properly working after the latest Meltano release.
• #1047 Fix a bug where some configuration values were not redacted
• #1069 Meltano will now use the schedule’s name to run incremental jobs
• #926 Move manual DigitalOcean Droplet configuration instructions to advanced tutorials
• Collapse Installation docs into a single section
• #1045 Make it clear that ‘meltano add’ is not hanging while installing plugins
• #1000 Update Getting Started guide with updated screenshots and content
• #854 Charts now use pretty labels rather than the ID
• #1011 Removed “Catch-up Date” in favor of default “Start Date” of extractor
• #578 Remove support for tap-zuora.
• #1002 Update discovery.yml with explicit kind: password metadata (we infer and set input types of password as a safeguard, but the explicit setting is preferred)
• #1049 Change default target-sqlite database name to warehouse to not conflict with system database
• #949 Update the way Meltano handles logs for ELT runs: Every elt run is logged in .meltano/run/logs/{job_id}/elt_{timestamp}.log. That allows Meltano to keep logs for multiple, or even concurrent, elt runs with the same job_id.
• #949 Update “Create Pipeline” redirect logic based on the previous route being ‘transforms’ (this is a UX win setting up the user with the sub-UI for the next logical step vs. requiring a manual “Create” click)
• #1051 Automatically set SQLALCHEMY_DATABASE_URI config to system database URI

Fixes

• #1173 Fix sortBy drag-and-drop bug in Analyze by properly using tryAutoRun vs. runQuery
• #1079 meltano elt will now run in isolation under .meltano/run/elt
• #1204 move project creation steps out of the local installation section of the docs and into the Getting Started Guide
• #782 Update timeframe label and fix timeframe attributes to properly display in the Result Table
• #1172 Fix analytics issue related to app version
• #1050 Fix a bug where the Job log would be created before the transform are run.
• #1122 meltano elt will now properly run when using target-snowflake.
• #1159 Minor UI fixes (proper MainNav Model icon active color during Analyze route match & “Run” auto query related cleanup) and ...NameFromRoute refactor renaming cleanup
• #1120 Fix a concurrency bug causing meltano select to crash.
• #1086 Fix a concurrency issue when the meltano.yml file was updated.
• #1112 Fix the “Run” button to improve UX by properly reflecting the running state for auto-running queries
• #1023 Fix last vuex mutation warning with editable localConfiguration clone approach
• Patch technicality due to PyPi limitation (v1 already existed from a publish mistake seven+ months ago) with needed changelog New/Changes/Fixes section headers
• #1071 Fix rehydratePollers so the UI reflects running jobs after a hard refresh or route change (this surfaced from the recent !963 change)
• #1075 Fix an issue where meltano elt would fail when a previous job was found
• #1004 Fix error when deselecting last attribute in Analyze
• #1048 Fix various actions that should have been mutations and did minor code convention cleanup
• #1063 Fix the “Explore” button link in Dashboards to properly account for the namespace
• #1085 Fix Analyze model dropdown to properly reflect installed models
• #1089 Properly re-initialize the Analyze page after a new analysis is selected during an existing analysis (this issue surfaced due to the recent Analyze dropdown CTAs addition which enables an analysis change during an existing one)
• #1092 Fix async condition so the design store’s defaultState is properly applied before loading a new design via initializeDesign

Breaks

• None

To see a full list of all improvements to Meltano at anytime, make sure to check out our CHANGELOG. Our release cadence is every Monday around 9am Pacific Time. Learn more about our release process.

I enjoyed a meeting run by my teammate Micael, and want to highlight it here to provide a window into how we work at Meltano. This video captures a meeting called by a senior engineer on our team to clarify the purpose and workflow around a feature. Through the discussion, it reveals other areas that need more attention and engages the rest of the team in dialoguing about who our ideal customers area, what they need, and how we can deliver.

You can watch us build Meltano in public through our YouTube channel, which is updated several times a week with tutorials, product speedruns and dogfooding, demo days showing recently completed work, and other ad hoc conversations like this one.

Because we are a fully remote team, working a-synchronously across several time zones from GTM -8 to GMT +2, video recordings of meetings and other important content is a huge part of how we work together. I have never done this before, and now I can’t imagine going back. Maybe you will be able to explore some of these tactics for leading your own team.

If you think this is cool, and want to work on Meltano, you can contact me at danielle@gitlab.com about our Senior Product Designer and Senior Product Manager roles — which we will be hiring for in early 2020.

Every Friday, the Meltano team gets together to showcase the work that was created that week. Check out what we did last week: