Now Available: Meltano v1.39.1

Today, we are excited to release Meltano version 1.39.1, which (among other things) adds various new settings that make it easier to customize Meltano UI and configure it for production:

To learn more about configuring Meltano UI for production, check out the brand new documentation

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.39.0 on July 8:

New

  • #2141 Add ui.session_cookie_domain setting, to be used instead of ui.server_name when hostname matching is not desirable.
  • #2142 Let discovery.yml manifest URL be overwritten using discovery_url setting.
  • #2083 Add ui.anonymous_readonly setting to allow anonymous users read-only access when ui.authentication is enabled.
  • #2108 Show icon with tooltip in UI when read-only mode is enabled
  • #2120 Add ui.logo_url setting to allow the customization of the logo shown in the top left of the Meltano UI interface.

Changes

  • #2140 Only have meltano ui complain about missing ui.server_name, ui.secret_key and ui.password_salt settings when ui.authentication is enabled
  • #2083 Stop allowing anonymous users read-only access when ui.authentication and ui.readonly are both enabled.

Fixes

  • #2151 Fix meltano ui never printing Meltano UI is now available at [URL] if ui.server_name is set
  • #2152 Fix meltano ui printing all gunicorn.error logs twice

Now Available: Meltano v1.39.0

Today, we are excited to release Meltano version 1.39.0, which (among other things) enables more streamlined and consistent configuration of all Meltano and Meltano UI settings using meltano config and the same highly flexible layered configuration model already used to manage plugin settings.

You can use meltano config meltano list (with meltano as the plugin_name) to list all available settings with their names, environment variables, and current values, or check out the brand-new settings documentation for more context and examples.

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.38.1 on July 3:

New

  • #2100 Let settings be stored directly in .env using meltano config set --store=dotenv
  • #2099 Let Meltano settings be configured using meltano config meltano {list,set,unset,reset}

Changes

  • #2100 Have meltano ui setup store secrets in .env instead of ui.cfg

Fixes

  • #2135 Fix UI “Explore” page for custom (as opposed to plugin-based) topics and models
  • #2136 Show error message in Analyze UI when pipeline for extractor is missing, even if extractor is installed
  • #2131 Have “true” environment variables take precedence over those defined in .env

Now Available: Meltano v1.38.1

Today, we are excited to release Meltano version 1.38.1, which (among other things) contains a significant 61-file refactor of how project settings and environment variables are managed internally.

These changes pave the way for more consistent and streamlined configuration of Meltano and Meltano UI using meltano config and the same highly flexible layered configuration model that is already used to manage plugin settings.

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.38.0 on June 30:

New

  • #2128 Allow alternative env vars to be specified for settings using env_aliases array
  • #2128 Allow negated env var to be defined for boolean settings using ! prefix in env_aliases array
  • #2129 Cast values of kind: integer settings set through environment variables
  • #2132 Support common WEB_CONCURRENCY env var to set meltano ui concurrency (workers)
  • #2091 Support common PORT env var to set meltano ui bind port

Changes

  • #2094 Turn “Connections” page into “Extractors” management UI
  • #2130 Have CLI respect MELTANO_PROJECT_ROOT env var when set instead of looking at current directory
  • #2131 Have “true” environment variables take precedence over those defined in .env

Now Available: Meltano v1.38.0

Today, we are excited to release Meltano version 1.38.0, which (among other things) helps you debug your plugin configuration by having meltano config <plugin> list show the current values of settings and where they come from (the environment, meltano.yml, the system database, or the default), alongside the setting names and their environment variables.

Additionally, custom settings like metadata.<entity>.<key> that have been added to meltano.yml manually or using meltano config <plugin> set <key> <value> are now listed as well, along with the environment variable names that can be used to override them.

Last but not least, meltano ui now no longer automatically starts the Airflow scheduler if the airflow orchestrator is installed. If you’d like to run the Airflow scheduler alongside the Meltano UI, you’ll have to start it manually using meltano invoke airflow scheduler.

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.37.0 and v1.37.1 on June 25:

New

  • #2122 Allow custom (meltano.ymlconfig-defined) settings to be overridden using environment
  • #2123 Show current values and their source in meltano config <plugin> list

Changes

  • #2102 Improve discovery.yml incompatibility handling and error messages

Fixes

  • #2121 Don’t include empty plugins object in new project meltano.yml

Breaks

  • #2105 Stop automatically running Airflow scheduler as part of Meltano UI

Now Available: Meltano v1.37.0 and v1.37.1

Today, we are excited to release Meltano version 1.37.0, which (among other things) lets you instantly containerize your Meltano project using meltano add files docker. Adding this file bundle to your project will add the Dockerfile and .dockerignore files needed to build a project-specific Docker container image containing Meltano, your project, and all of its plugins.

If you’d like to use GitLab CI/CD to continuously build your Meltano project’s Docker image and push it to GitLab’s built-in Container Registry, you can add the appropriate .gitlab-ci.yml and .gitlab/ci/docker.gitlab-ci.yml files to your project by adding the gitlab-ci file bundle.

Learn more about containerizing your Meltano project and deployment in production.


Shortly after v1.37.0 was released, user Charles Julian Knight reported that one of the included changes introduced a bug that caused meltano invoke airflow to fail. This has been fixed in Meltano version 1.37.1. Thanks for reporting this issue, Julian!

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.36.1 on June 19:

New

  • #2048 Add docker and gitlab-ci file bundles to allow instant containerization
  • #2068 Add interface to schedule new pipelines to Pipelines page
  • #2081 Add url config option to target-postgres
  • #2060 Add --format=json option to meltano schedule list so that a project’s schedules can be processed programmatically

Now Available: Meltano v1.36.1

Today, we are excited to release Meltano version 1.36.1, which (among other things) adds the ability to specify Singer tap stream and property metadata (like replication-method and replication-key) as if it were any other plugin configuration, using special nested config properties metadata.<entity>.<key> and metadata.<entity>.<attribute>.<key>:

meltano config <plugin_name> set metadata <entity> <key> <value>
meltano config <plugin_name> set metadata <entity> <attribute> <key> <value>

# For example:
meltano config tap-postgres set metadata "some_schema-*" replication-method INCREMENTAL
meltano config tap-postgres set metadata "some_schema-*" replication-key created_at

meltano config tap-postgres set metadata some_schema-some_table some_column custom-metadata custom-value

Whenever an extractor is run using meltano elt or meltano invoke, Meltano will generate the desired catalog on the fly by running the tap in discovery mode and applying the selection and metadata rules to the resulting catalog file before passing it to the tap in sync mode.

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.36.0 on June 15:

New

  • #2067 Add pipeline name, loader, and transform columns to Pipelines table
  • #2103 Support nested objects in meltano.yml config objects
  • #2103 Allow nested properties to be set using meltano config by specifying a list of property names: meltano config <plugin_name> set <property> <subproperty> <value>
  • #2026 Allow Singer stream and property metadata to be configured using special nested config properties metadata.<entity>.<key> and metadata.<entity>.<attribute>.<key>.

Fixes

  • #2102 Fix potential meltano upgrade failures by having it invoke itself with --skip-package after upgrading the package to ensure it always uses the latest code.

Now Available: Meltano v1.36.0

Today, we are excited to release Meltano version 1.36.0, which (among other things) introduces “file bundle” plugins that are used to automatically add plugin-specific files to your Meltano project when adding a plugin like the dbt transformer or Airflow orchestrator.

Before, these Airflow and dbt-specific files would be included in every newly initialized Meltano project, even if neither plugin was never actually used.

These changes pave the way for resolving the “Add Dockerfile to new project that will build project-specific image with plugins already installed” issue with a new docker file bundle that can easily be added to any Meltano project using meltano add files docker.

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.35.1 on June 11:


New

  • #2095 Support singular and plural plugin types in CLI arguments
  • #2095 Allow list of plugin names of same type to be provided to meltano add and meltano install
  • #2089 Add new ‘files’ plugin type to allow plugin-specific files to be added to a Meltano project automatically
  • #2090 Allow SERVER_NAME, SECRET_KEY, and SECURITY_PASSWORD_SALT to be set using environment instead of ui.cfg

Fixes

  • #2096 Remove noisy migration-related logging from CLI command output
  • #2089 Don’t add Airflow and dbt-specific files to Meltano project until plugins are added explicitly
  • #2089 Don’t add docker-compose.yml to Meltano project by default

Now Available: Meltano v1.35.1

Today, we are excited to release Meltano version 1.35.1, which fixes 2 minor bugs and makes the output of meltano add and meltano install consistent in how they refer to plugin types and names.

These changes pave the way for the addition of a new files plugin type, which will allow us to resolve the “Add Dockerfile to new project that will build project-specific image with plugins already installed” issue with a new meltano add files docker command.

Excited to try it out?

To upgrade your local installation of Meltano, activate the appropriate Python virtual environment and run meltano upgrade from inside a Meltano project, or pip3 install --upgrade meltano from anywhere else. If you’re running Meltano inside Docker, run docker pull meltano/meltano.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.35.0 on June 9:

Changes

  • #2094 Consistently refer to plugin types and names in CLI output

Fixes

  • #2092 Only install plugins related to plugins of the specified type when running meltano install <plugin_type> --include-related
  • #2093 Print error when transform plugin is installed before dbt

Building Meltano in Public: four weeks into the relaunch

Earlier this week, it was my turn to host a GitLab Group Conversation (a publicly live streamed Q&A on the GitLab Unfiltered YouTube channel) on Meltano. I used the opportunity to share some updates about last month’s announcement of Meltano’s new strategy and direction and the response and impact so far.

If you’re curious, check out the presentation on Google Slides and the Q&A on YouTube. The presentation content is also reproduced below, as is an embedded video of the Q&A.

Personally, I couldn’t be happier with how things have turned out so far, and I’m incredibly grateful to everyone who has given Meltano a try, shared feedback from their experience, and contributed to the future of this open source ELT solution.

In conclusion, allow me to quote from the Slack conversation I had with GitLab CEO Sid Sijbrandij after he saw the presentation:

  • Sid: great content, love the quotes on slide 3, these people really get what you’re doing.
  • Douwe: Yep, and attracting a few dozen people who really get it, will stick with it, and will help make it happen is exactly what I was aiming for, so I’d call this a successful relaunch! 😁

Group Conversation Presentation

Getting the word out about Meltano’s new direction

In the previous Meltano GC, I shared drafts of 2 blog posts announcing Meltano’s new strategy and direction. These were published on the Meltano blog on May 13:

  1. Revisiting the Meltano strategy: a return to our roots
    1. Key phrase: We will go all-in on positioning Meltano as an open source self-hosted platform for running data integration and transformation (ELT) pipelines.
  2. Why we are building an open source platform for ELT pipelines
    1. Key phrase: Our goal is to make the power of data integration available to all by building a true open source alternative to existing proprietary hosted ELT solutions.

On May 18, I got a chance to talk about this new direction on the Drill to Detail podcast: ‘Meltano, Singer Taps and Open-Source Data Pipelines’ with Special Guest Douwe Maan

The goal: attract existing Singer community members, get them to give Meltano a try, gather feedback on how it currently stacks up as an ELT pipeline platform, and acquire contributors

The response has been overwhelmingly positive

Blog post 2 reached the frontpage of the r/dataengineering subreddit, with very positive responses: (emphasis mine)

  • So if I understand it correctly the project is about building an open source community to build and maintain an array of SaaS APIs so people can spend less or no time getting that data, for free?
    Seems brilliant and wonder why it hasn’t been done so far… Hell, just sorting out Facebook Ads and Google Campaign Manager APIs could be a game changer for a lot of smaller advertising agencies who otherwise can never afford the tools or expertise to do these in house.
    Will be following closely and would love to maybe even try and contribute at some point!
  • This is very exciting, I like the standing on the shoulders of giants approach, by sitting in top of existing leading open source initiatives in this space like airflow, dbt and singer, and making them easier and faster to coordinate and code with. I’ll be following this.

Responses on Twitter have been similarly positive:

  • This sounds terrific! Really hope it can reinvigorate the Singer community… so far I’ve been really surprised by how much of a ghost town most taps are. This seems like it could be a great catalyst.
  • I LOVE LOVE LOVE the new @meltanodata. This is what the future of the world’s data infrastructure looks like. If @stitch_data and @singer_io are the gateway drug… @meltanodata is the complete thing.

Impact by the numbers

  • 3845 visitors to meltano.com between May 13, 14, and 15, where we normally would have seen about 100 per day (up 1182% from 300)
  • Since the announcement:
    • 55 new Twitter followers (up 4% from 1469)
    • 64 new Slack workspace members (up 23% from 273)
    • 351 new users of the CLI, where we normally would have seen about 170 (up 107%)
  • 103 weekly active members on the Slack workspace 3 weeks after the announcement, up from 38 just before (up 171%)
  • 6 new issues by and with comments from 6 people (excluding me) that have allowed me to flesh out and start working on the Running pipelines in production epic
  • 5 new MRs by 3 people (excluding me)
  • 6 new releases of Meltano with 15 bug fixes and many changes and new features, most in response to feedback from new users

Other positive outcomes

  • I’ve been invited to talk about Meltano on The Data Engineering Podcast and will be interviewed on June 15
  • RFA’s Applied Labs are considering replacing PipelineWise with Meltano in the AppliedData platform they offer to clients, and are planning to put 1.5 developers on Meltano for the next 2 months to work on the Data pipeline management UI epic
  • Multiple other data engineering consultants and consultancies are evaluating Meltano and considering adopting it as part of the stack they set up for clients
  • A commercial player in the hosted data pipeline orchestration space has expressed interest in finding ways to collaborate
  • A few GitLab customers have informed about Meltano through their account managers

Epics for current priorities

Group Conversation Q&A

Now Available: Meltano v1.35.0

Today, we are excited to release Meltano version 1.35.0, which (among other things) makes it easier to manage and version control a Meltano project’s configuration by having meltano config store values in meltano.yml by default, instead of the system database.

Sensitive and environment-specific values can still be stored in the system database via the UI or meltano config <plugin> set --store=db <key> <value>.

Learn more about managing the configuration of your plugins

Excited to try it out?

To upgrade your local installation of Meltano, activate the appropriate Python virtual environment and run meltano upgrade from inside a Meltano project, or pip3 install --upgrade meltano from anywhere else. If you’re running Meltano inside Docker, run docker pull meltano/meltano.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.34.2 on May 29:

New

  • #2013 Add --store option to meltano config with possible values db and meltano_yml
  • #2087 Add --plugin-type option to meltano config and meltano invoke
  • #2088 Add meltano upgrade subcommands package, files, database, and models

Changes

  • #2064 Print environment-specific instructions when meltano upgrade is run from inside Docker
  • #2013 Have meltano config store in meltano.yml instead of system database by default
  • #2087 Skip plugins that are not configurable or invokable when finding plugin by name in meltano config and meltano invoke

Fixes

  • #2080 Don’t try to overwrite .gitignore when upgrading Meltano and project
  • #2065 Don’t have meltano upgrade complain when meltano ui‘s gunicorn isn’t running
  • #2085 Don’t change order of object and set values when meltano.yml is updated programatically
  • #2086 Ensure “meltano config –format=json” prints actual JSON instead of Python object