To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.
What else is new?
The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.39.0 on July 8:
New
#2141 Add ui.session_cookie_domain setting, to be used instead of ui.server_name when hostname matching is not desirable.
#2142 Let discovery.yml manifest URL be overwritten using discovery_url setting.
#2083 Add ui.anonymous_readonly setting to allow anonymous users read-only access when ui.authentication is enabled.
#2108 Show icon with tooltip in UI when read-only mode is enabled
#2120 Add ui.logo_url setting to allow the customization of the logo shown in the top left of the Meltano UI interface.
Changes
#2140 Only have meltano ui complain about missing ui.server_name, ui.secret_key and ui.password_salt settings when ui.authentication is enabled
#2083 Stop allowing anonymous users read-only access when ui.authentication and ui.readonly are both enabled.
Fixes
#2151 Fix meltano ui never printing Meltano UI is now available at [URL] if ui.server_name is set
#2152 Fix meltano ui printing all gunicorn.error logs twice
You can use meltano config meltano list (with meltano as the plugin_name) to list all available settings with their names, environment variables, and current values, or check out the brand-new settings documentation for more context and examples.
Excited to try it out?
To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.
What else is new?
The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.38.1 on July 3:
New
#2100 Let settings be stored directly in .env using meltano config set --store=dotenv
#2099 Let Meltano settings be configured using meltano config meltano {list,set,unset,reset}
Changes
#2100 Have meltano ui setup store secrets in .env instead of ui.cfg
Fixes
#2135 Fix UI “Explore” page for custom (as opposed to plugin-based) topics and models
#2136 Show error message in Analyze UI when pipeline for extractor is missing, even if extractor is installed
#2131 Have “true” environment variables take precedence over those defined in .env
To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.
What else is new?
The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.38.0 on June 30:
New
#2128 Allow alternative env vars to be specified for settings using env_aliases array
#2128 Allow negated env var to be defined for boolean settings using ! prefix in env_aliases array
#2129 Cast values of kind: integer settings set through environment variables
#2132 Support common WEB_CONCURRENCY env var to set meltano ui concurrency (workers)
#2091 Support common PORT env var to set meltano ui bind port
Changes
#2094 Turn “Connections” page into “Extractors” management UI
#2130 Have CLI respect MELTANO_PROJECT_ROOT env var when set instead of looking at current directory
#2131 Have “true” environment variables take precedence over those defined in .env
Today, we are excited to release Meltano version 1.38.0, which (among other things) helps you debug your plugin configuration by having meltano config <plugin> list show the current values of settings and where they come from (the environment, meltano.yml, the system database, or the default), alongside the setting names and their environment variables.
Last but not least, meltano ui now no longer automatically starts the Airflow scheduler if the airflow orchestrator is installed. If you’d like to run the Airflow scheduler alongside the Meltano UI, you’ll have to start it manually using meltano invoke airflow scheduler.
Excited to try it out?
To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.
What else is new?
The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.37.0 and v1.37.1 on June 25:
New
#2122 Allow custom (meltano.yml–config-defined) settings to be overridden using environment
#2123 Show current values and their source in meltano config <plugin> list
Changes
#2102 Improve discovery.yml incompatibility handling and error messages
Fixes
#2121 Don’t include empty plugins object in new project meltano.yml
Breaks
#2105 Stop automatically running Airflow scheduler as part of Meltano UI
Today, we are excited to release Meltano version 1.37.0, which (among other things) lets you instantly containerize your Meltano project using meltano add files docker. Adding this file bundle to your project will add the Dockerfile and .dockerignore files needed to build a project-specific Docker container image containing Meltano, your project, and all of its plugins.
If you’d like to use GitLab CI/CD to continuously build your Meltano project’s Docker image and push it to GitLab’s built-in Container Registry, you can add the appropriate .gitlab-ci.yml and .gitlab/ci/docker.gitlab-ci.yml files to your project by adding the gitlab-ci file bundle.
To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.
What else is new?
The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.36.1 on June 19:
New
#2048 Add docker and gitlab-ci file bundles to allow instant containerization
#2068 Add interface to schedule new pipelines to Pipelines page
To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.
What else is new?
The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.36.0 on June 15:
New
#2067 Add pipeline name, loader, and transform columns to Pipelines table
#2103 Support nested objects in meltano.ymlconfig objects
#2103 Allow nested properties to be set using meltano config by specifying a list of property names: meltano config <plugin_name> set <property> <subproperty> <value>
#2026 Allow Singer stream and property metadata to be configured using special nested config properties metadata.<entity>.<key> and metadata.<entity>.<attribute>.<key>.
Fixes
#2102 Fix potential meltano upgrade failures by having it invoke itself with --skip-package after upgrading the package to ensure it always uses the latest code.
Today, we are excited to release Meltano version 1.36.0, which (among other things) introduces “file bundle” plugins that are used to automatically add plugin-specific files to your Meltano project when adding a plugin like the dbt transformer or Airflow orchestrator.
Before, these Airflow and dbt-specific files would be included in every newly initialized Meltano project, even if neither plugin was never actually used.
To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.
What else is new?
The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.35.1 on June 11:
New
#2095 Support singular and plural plugin types in CLI arguments
#2095 Allow list of plugin names of same type to be provided to meltano add and meltano install
#2089 Add new ‘files’ plugin type to allow plugin-specific files to be added to a Meltano project automatically
#2090 Allow SERVER_NAME, SECRET_KEY, and SECURITY_PASSWORD_SALT to be set using environment instead of ui.cfg
Fixes
#2096 Remove noisy migration-related logging from CLI command output
#2089 Don’t add Airflow and dbt-specific files to Meltano project until plugins are added explicitly
#2089 Don’t add docker-compose.yml to Meltano project by default
Today, we are excited to release Meltano version 1.35.1, which fixes 2 minor bugs and makes the output of meltano add and meltano install consistent in how they refer to plugin types and names.
Earlier this week, it was my turn to host a GitLab Group Conversation (a publicly live streamed Q&A on the GitLab Unfiltered YouTube channel) on Meltano. I used the opportunity to share some updates about last month’s announcement of Meltano’s new strategy and direction and the response and impact so far.
If you’re curious, check out the presentation on Google Slides and the Q&A on YouTube. The presentation content is also reproduced below, as is an embedded video of the Q&A.
Personally, I couldn’t be happier with how things have turned out so far, and I’m incredibly grateful to everyone who has given Meltano a try, shared feedback from their experience, and contributed to the future of this open source ELT solution.
In conclusion, allow me to quote from the Slack conversation I had with GitLab CEO Sid Sijbrandij after he saw the presentation:
Sid: great content, love the quotes on slide 3, these people really get what you’re doing.
Douwe: Yep, and attracting a few dozen people who really get it, will stick with it, and will help make it happen is exactly what I was aiming for, so I’d call this a successful relaunch! 😁
Group Conversation Presentation
Getting the word out about Meltano’s new direction
In the previous Meltano GC, I shared drafts of 2 blog posts announcing Meltano’s new strategy and direction. These were published on the Meltano blog on May 13:
Key phrase: We will go all-in on positioning Meltano as an open source self-hosted platform for running data integration and transformation (ELT) pipelines.
Key phrase: Our goal is to make the power of data integration available to all by building a true open source alternative to existing proprietary hosted ELT solutions.
The goal: attract existing Singer community members, get them to give Meltano a try, gather feedback on how it currently stacks up as an ELT pipeline platform, and acquire contributors
So if I understand it correctly the project is about building an open source community to build and maintain an array of SaaS APIs so people can spend less or no time getting that data, for free? Seems brilliant and wonder why it hasn’t been done so far… Hell, just sorting out Facebook Ads and Google Campaign Manager APIs could be a game changer for a lot of smaller advertising agencies who otherwise can never afford the tools or expertise to do these in house. Will be following closely and would love to maybe even try and contribute at some point!
This is very exciting, I like the standing on the shoulders of giants approach, by sitting in top of existing leading open source initiatives in this space like airflow, dbt and singer, and making them easier and faster to coordinate and code with. I’ll be following this.
Responses on Twitter have been similarly positive:
This sounds terrific! Really hope it can reinvigorate the Singer community… so far I’ve been really surprised by how much of a ghost town most taps are. This seems like it could be a great catalyst.
I LOVE LOVE LOVE the new @meltanodata. This is what the future of the world’s data infrastructure looks like. If @stitch_data and @singer_io are the gateway drug… @meltanodata is the complete thing.
Impact by the numbers
3845 visitors to meltano.com between May 13, 14, and 15, where we normally would have seen about 100 per day (up 1182% from 300)
RFA’s Applied Labs are considering replacing PipelineWise with Meltano in the AppliedData platform they offer to clients, and are planning to put 1.5 developers on Meltano for the next 2 months to work on the Data pipeline management UI epic
Multiple other data engineering consultants and consultancies are evaluating Meltano and considering adopting it as part of the stack they set up for clients
A commercial player in the hosted data pipeline orchestration space has expressed interest in finding ways to collaborate
A few GitLab customers have informed about Meltano through their account managers
Today, we are excited to release Meltano version 1.35.0, which (among other things) makes it easier to manage and version control a Meltano project’s configuration by having meltano config store values in meltano.yml by default, instead of the system database.
Sensitive and environment-specific values can still be stored in the system database via the UI or meltano config <plugin> set --store=db <key> <value>.
