Today, we are excited to release Meltano version 1.36.1, which (among other things) adds the ability to specify Singer tap stream and property metadata (like replication-method
and replication-key
) as if it were any other plugin configuration, using special nested config
properties metadata.<entity>.<key>
and metadata.<entity>.<attribute>.<key>
:
meltano config <plugin_name> set metadata <entity> <key> <value>
meltano config <plugin_name> set metadata <entity> <attribute> <key> <value>
# For example:
meltano config tap-postgres set metadata "some_schema-*" replication-method INCREMENTAL
meltano config tap-postgres set metadata "some_schema-*" replication-key created_at
meltano config tap-postgres set metadata some_schema-some_table some_column custom-metadata custom-value
Whenever an extractor is run using meltano elt
or meltano invoke
, Meltano will generate the desired catalog on the fly by running the tap in discovery mode and applying the selection and metadata rules to the resulting catalog file before passing it to the tap in sync mode.
Excited to try it out?
To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade
. This will upgrade the meltano
package and apply any necessary changes to your project.
What else is new?
The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.36.0 on June 15:
New
- #2067 Add pipeline name, loader, and transform columns to Pipelines table
- #2103 Support nested objects in
meltano.yml
config
objects - #2103 Allow nested properties to be set using
meltano config
by specifying a list of property names:meltano config <plugin_name> set <property> <subproperty> <value>
- #2026 Allow Singer stream and property metadata to be configured using special nested
config
propertiesmetadata.<entity>.<key>
andmetadata.<entity>.<attribute>.<key>
.
Fixes
- #2102 Fix potential
meltano upgrade
failures by having it invoke itself with--skip-package
after upgrading the package to ensure it always uses the latest code.