Meltano 3.0

Meltano is growing up! We just shipped the 3rd major version of Meltano, including many changes.

Let’s look at them in detail!

TL;DR: Look into the migration guide! The UI isn’t available anymore, Lock files are required for better production deployments, Postgres backends now need to be installed with extra components, and the “target_schema” setting was removed. You can now directly invoke mappers!

Let’s look into the details.

Big breaking changes

Meltano shines as a YAML-based and CLI-first data integration engine. Therefore, we’ve made the decision to deprecate the old UI already in v2.12.0. 

With 3.0, we’ve now deleted the UI and the accompanying API.

The meltano discover command was also removed because few people were using it, and we’d like to focus our efforts on the hub as the source of truth for all Meltano connectors. We heard feedback that some users used that command to validate if the Hub API was available, so to make that easier, we added a

meltano hub ping 

command.

The final change towards even better production deployment is the requirement of lock files. When you run the

meltano install

command, the definition of a plugin is retrieved from the hub. This specific version is then stored as a lock file because, as you might imagine, the hub definition gets updated and might change later on. So, a lock file is the only option to truly version control your dependencies.

But missing lock files cause problems during production deployments (that often do not have access to retrieve missing lock files again), so from now on, they are required.

New cool features 

We also added a bunch of new cool features.

• meltano el as meltano elt alias
• You can now set a python version for each plugin
• You can now add mappers into the meltano invoke command. You can use them by piping inputs to them (e.g. meltano invoke tap-gitlab | meltano invoke remove-gitlab-pii)
• We now have a meltano hub ping command
• We got a meltano config –unsafe flag — Thanks @ReubenFrankel!
• Support psycopg[binary] under a new postgres extra
• Bump SQLAlchemy to 2.0
• Add “did you mean” CLI command name suggestions
• Add aliases for the @once schedule interval

For the full list of changes, check out the release notes; thanks as always to the community contributors on this release!