Building Meltano in Public: November 2019 in Review

In November, we released Meltano as a free hosted product, eliminating the hosting and installation hassle with a concierge setup so you can start building dashboards faster.


186% Month-over-Month Increase in Meltano Usage

Our goal is to grow Meltano MAUI (monthly active UI users) by 10% week-over-week, which is similar to the growth metric we are challenged to achieve while in Y Combinator. At the target pace (in red) we will have 1,000 MAUI by the end of 2019. While we experienced significant month-over-month growth of MAUI relative to October, we are still behind pace.

See last month’s growth graph.


Contributions & Cadence

Thank you to all our contributors, both inside GitLab and across the broader open source community for Meltano. Get involved with the open source Meltano project by reading our Contributor Guide and joining the discussion in the public Meltano Slack community.

MRs Merged by Month

Top Contributors by Opened MR Count


Consolidated CHANGELOG

New

  • #1222 Include static application security testing (SAST) in the pipeline
  • #1164 Add “transform limitations” message to Transform UI
  • #1272 Add Vuepress plugin to generate a sitemap on website build
  • meltano-marketing#89 Adds basic title and meta descriptions to all public-facing website & documentation pages.
  • #1235 Add help link button in the app
  • #1285 Add link to YouTube guidelines for release instructions
  • #1277 Move sections that don’t apply to outside contributors from Contributing and Roadmap docs to Handbook: Release Process, Release Schedule, Demo Day, Speedruns, DigitalOcean Marketplace
  • #1236 Add contextual Analyze CTAs in the Job Log UI
  • #764 Add plugin profiles to enable multiple configurations for extractors
  • #1081 Add ability to delete data pipelines
  • #1217 Add “Test Connection” button to validate connection settings prior to ELT runs
  • #1236 Add contextual Analyze CTAs in the Job Log UI
  • #1271 Add labels in discovery.yml for easy brand definition

Changes

  • #1239 Update header buttons layout on small viewports
  • #1019 Automatically update package.json file versions
  • #1253 Do not allow meltano command invocation without any argument
  • #1192 Improve helper notes associated with each Extract, Load, and Transform step to better communicate the purpose of each
  • #1201 Improved “Auto Advance” messaging regarding Entity Selection. We also doubled the default toast time to improve likelihood of reading feedback.
  • #1191 update Google Analytics extractor documentation to explain how to set up the Google Analytics API, and remove duplicate instructions from the Google Analytics API + Postgres tutorial
  • #1199 Add example and sample CSV files to the CSV extractor documentation
  • #1247 Update the Loading CSV Files to a Postgres Database Tutorial to use Meltano UI for setting up the Extractor and Loader, running the ELT pipeline and analyzing the results. Also provide all the files used in the tutorial (transformations, models, etc) as downloadable files.
  • #1279 Revise “Roadmap” section of the docs with clarified persona, mission, vision, and re-order content
  • #1134 Update the GitLab API + Postgres. Include video walk-through and update the end to end flow to only use Meltano UI.
  • #95 Update the DigitalOcean CTA to go to the public directory page for the Meltano droplet
  • #1270 Main navigation “Pipeline” to “Pipelines” to reinforce multiple vs. singular (conflicts a bit with the verb approach of the other navigation items but we think it’s worth it for now)
  • #1240 Provide clarity around how Airflow can be used directly in documentation and UI
  • #1263 Document lack of Windows support and suggest WSL, Docker
  • #1257 Prevent modified logo file upon each build
  • #1289 Dismiss all modals when using the escape key
  • #1282 Remove Entity Selection from the UI (still available in CLI) and default to “All” entities for a given data source
  • #1303 Update the configuration options for the Salesforce Extractor to only include relevant properties. Remove properties like the client_id that were not used for username/password authentication.
  • #1308 Update the configuration options for the Marketo Extractor to use a Start Date instead of a Start Time.
  • #1323 Add CTA to send users to Typeform to provide info for setting up a hosted dashboard
  • #1271 Improve messaging on tap and target settings modals
  • #1226 Update Pipelines main navigation link to show all data pipeline schedules if that step has been reached
  • #1323 Add CTA to send users to Typeform to provide info for setting up a hosted dashboard
  • #1271 Improve messaging on tap and target settings modals
  • #1246 Update the Salesforce API + Postgres Tutorial to use Meltano UI for setting up the Extractor and Loader, running the ELT pipeline and analyzing the results.
  • #1225 Update dbt docs link to be conditional so the user doesn’t experience 404s

Fixes

  • #1259 Fix meltano elt not properly logging errors happening in the ELT process
  • #1183 Fix a race condition causing the meltano.yml to be empty in some occurence
  • #1258 Fix format of custom extractor’s capabilities in meltano.yml
  • #1215 Fix intercom documentation footer overlap issue.
  • #1215 Fix YouTube iframes to be responsive (resolves unwanted side-effect of horizontal scrollbar at mobile/tablet media queries)
  • #1297 Get actual latest ELT job log by sorting matches by creation time with nanosecond resolution
  • #1297 Fix pipeline failure caused by jobs that require true concurrency being executed on CI runners that don’t
  • #1298 Let default entity selection be configured in discovery.yml under select
  • #1298 Define default entity selection for tap-salesforce
  • #1304 Fix Meltano subprocess fetching large catalogs (e.g. for Salesforce) getting stuck do to the subprocess’ stderr buffer filling and the process getting deadlocked.
  • #1184 Fix contextualModels implementation for contextual CTAs in Job Log modal
  • #1318 Pin dbt version to v0.14.4 to address Meltano Transformation failing when using dbt v0.15.0

As Meltano progresses in maturity, functionality and features and the team continues to grow we are publishing more MRs, updates, and videos than ever before. Starting this month, we will recap our progress here and through the Meltano newsletter.

Published by Danielle Morrill

General Manager of Meltano at GitLab

Leave a comment

Your email address will not be published. Required fields are marked *