Now Available: Meltano v1.41.1

Today, we are excited to release Meltano version 1.41.1, which (among other things) lets you easily override Singer stream schema descriptions for specific attributes (e.g. {"type": ["string", "null"], "format": "date-time"}) using a new schema extractor extra:

# In `meltano.yml`
extractors:
- name: tap-postgres
  pip_url: tap-postgres
  schema:
    some_table:
      created_at:
        type: ["string", "null"]
        format: date-time

This plugin extra can also be thought of and interacted with as a setting named _schema, with _schema.<entity>.<attribute> and _schema.<entity>.<attribute>.<key> nested properties:

# On the command line
meltano config <plugin> set _schema <entity> <attribute> <schema description>
meltano config <plugin> set _schema <entity> <attribute> <key> <value>

export <NAMESPACE>__SCHEMA='{"<entity>": {"<attribute>": {"<key>": "<value>"}}}'

# Once schema descriptions have been set in `meltano.yml`, 
# environment variables can be used to override specific
# nested properties:
export <NAMESPACE>__SCHEMA_<ENTITY>_<ATTRIBUTE>_<KEY>=<value>

# For example:
meltano config tap-postgres set _metadata some_table created_at type '["string", "null"]'
meltano config tap-postgres set _metadata some_table created_at format date-time

export TAP_POSTGRES__SCHEMA_SOME_TABLE_CREATED_AT_FORMAT=date

Whenever an extractor is run using meltano elt or meltano invoke, Meltano will generate the desired catalog on the fly by running the tap in discovery mode and applying the selection, metadata, and schema rules to the resulting catalog file before passing it to the tap in sync mode.

Check out the brand-new documentation for more about plugin extras in general and the select, metadata, and schema extractor extras in particular.

---

A special thanks goes out to Dmitry Stadnik of RFA, who contributed an improvement to Meltano UI: List installed and available extractors separately on Extractors page.

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.41.0 on July 20:

New

• #2196 Pretty print and apply appropriate nesting to meltano config output
• #2003 Let extractor extra select be interacted with as _select setting
• !1774 Let transform extra vars be interacted with as _vars setting
• !1774 Add support for kind: object settings, which can gather nested values from across setting stores
• !1774 Add support for aliases: [...] on setting definitions
• !1774 Add support for value_processor: 'nest_object' on setting definitions
• #2145 Let discovered catalog schema be modified using schema rules stored in extractor schema extra (aka _schema setting)

Changes

• #2070 List installed and available extractors separately on Extractors page
• !1774 Replace update.* file bundle settings with update extra (aka _update setting)
• !1774 Replace metadata.* extractor settings with metadata extra (aka _metadata setting)
• #1764 Interpret meltano config <plugin> set value as JSON to allow non-string values to be set more easily

Fixes

• !1774 Fix poor performance of setting lookups using more memoization