Today, we are excited to release Meltano version 1.50.0, which lets you manually provide extractor catalog and state files to meltano elt using new --catalog and --state options (and catalog and state extras), as an alternative to letting the catalog be generated on the fly and letting state be looked up based on the Job ID.

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.49.0 on September 15:

New

• #2291 Add catalog extractor extra to allow a catalog to be provided manually
• #2291 Add --catalog option to meltano elt to allow a catalog to be provided manually
• #2289 Add state extractor extra to allow state file to be provided manually
• #2289 Add --state option to meltano elt to allow state file to be provided manually

Fixes

• #2352 meltano elt --select and --exclude no longer unexpectedly select entities for extraction that match the wildcard pattern but weren’t selected originally.

Today, we are excited to release Meltano version 1.49.0, which (among other things) standardizes on <PLUGIN_NAME>_<SETTING_NAME> for configuration environment variable names (e.g. TARGET_POSTGRES_HOST for target-postgres‘s host setting instead of PG_ADDRESS), makes environment variable expansion in setting values more flexible, and uses this to let you easily override your extractor’s load_schema.

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.48.0 on September 7:

New

• #2279 Populate primary setting env var and aliases when invoking plugin or expanding env vars
• #2349 Allow plugin setting (default) values to reference pipeline plugin extras using generic env vars, e.g. MELTANO_EXTRACT__<EXTRA>
• #2281 Allow plugin extra (default) values to reference plugin name, namespace, profile using generic env vars, e.g. MELTANO_EXTRACTOR_NAMESPACE
• #2280 Allow plugin extra (default) values to reference plugin settings using env vars, e.g. target_schema: $PG_SCHEMA
• #2278 Read setting values from <PLUGIN_NAME>_<SETTING_NAME> env vars, taking precedence over <PLUGIN_NAMESPACE>_<SETTING_NAME> but not custom env
• #2350 Add MELTANO_TRANSFORM_* transform pipeline env vars for transformer (configuration) to access
• #2282 Add new extractor extra load_schema and use it as default loader schema instead of namespace
• #2284 Add new loader extra dialect and use it as default dbt target and Meltano UI SQL dialect instead of namespace
• #2283 Add new loader extra target_schema and use it as default dbt source_schema instead of loader schema
• #2285 Add new transform extra package_name and use it in dbt’s dbt_project.yml and --models argument instead of namespace

Changes

• #2279 Fall back on setting values from <PLUGIN_NAME>_<SETTING_NAME> and <PLUGIN_NAMESPACE>_<SETTING_NAME> env vars if a custom env is defined but not used
• #2278 Stop unnecessarily prepopulating env on a newly added custom plugin’s settings definitions
• #2208 Standardize on setting env vars prefixed with plugin name, not namespace or custom env

Today, we are excited to release Meltano version 1.48.0, which (among other things) lets you extract a subset of selected entities using new --select and --exclude options on meltano elt, improves plugin invocation and extractor catalog discovery error messages, and changes where meltano elt logs and generated plugin config files are stored.

Users who were depending on the previous locations of meltano elt logs ( .meltano/run/elt/<job ID>/<run ID>/elt.log) and extractor catalog files (.meltano/run/<extractor>/tap.properties.json) will want to use the new locations instead: .meltano/logs/elt/<job ID>/<run ID>/elt.log and .meltano/extractors/<extractor>/tap.properties.json.

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.47.0 on September 3:

New

• #2340 Print full error message when extractor catalog discovery fails
• #2223 Print clear error message when meltano invoke or meltano elt attempts to execute a plugin that hasn’t been installed yet
• #2345 Make tap-csv files a known setting
• #2155 Add select_filter extractor extra to allow extracting a subset of selected entities
• #2155 Add --select and --exclude options to meltano elt to allow extracting a subset of selected entities

Changes

• #2167 Include extractor and loader name in autogenerated meltano elt job ID
• #2343 Automatically delete generated plugin config files at end of meltano elt and meltano invoke

Fixes

• #2167 Make sure autogenerated meltano elt job ID matches in system database and .meltano/{run,logs}/elt
• #2347 Have meltano config <plugin> set --store=dotenv store valid JSON values for arrays and objects
• #2346 Correctly cast environment variable string value when overriding custom array and object settings

Breaks

• #2344 Move meltano elt output logs from .meltano/run/elt to .meltano/logs/elt
• #2342 Store pipeline-specific generated plugin config files (tap.config.json, tap.properties.json, etc) under .meltano/run/elt/<job_id>/<run_id> instead of .meltano/run/<plugin_name>. Users who were explicitly putting a catalog file at .meltano/run/<plugin_name>/tap.properties.json should use .meltano/extractors/<plugin_name>/tap.properties.json instead.

Today, we are excited to release Meltano version 1.47.0, which adds support for Bing Ads (thanks, VividFront!), prints docs and repo URLs when adding plugins, lets you specify a full schema for taps that can’t discover theirs, automatically uppercases target-snowflake‘s schema setting (thanks, Nevin Morgan!), and fixes a bug with embedded reports in the UI (thanks, Allan Whatmough!).

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.46.0 on August 27:

New

• #2210 Print documentation and repository URLs when adding a new plugin to the project
• #2277 Add tap-bing-ads as a known extractor
• #2328 Add new upcase_string setting value_processor that will convert string values to uppercase

Changes

• #2216 Add stream properties defined in an extractor’s schema extra to catalog if they do not exist in the discovered stream schema yet

Fixes

• #2338 Once again change target-csv to use singer-io/target-csv instead of the Meltano fork
• #2235 Make embed links accessible when not authenticated
• #2328 Always convert target-snowflake schema setting value to uppercase before passing it to plugin

Today, we are excited to release Meltano version 1.46.0, which (among other things) brings out-of-the-box support for extractor tap-spreadsheets-anywhere (docs), which lets you pull data from CSV and Excel files stored locally or in the cloud (S3, SFTP, HTTP, etc):

meltano add extractor tap-spreadsheets-anywhere

To learn more about tap-spreadsheets-anywhere, read the docs and the README.

Special thanks go out to Eric Simmerman for building this tap and contributing support to Meltano!

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.45.0 on August 17:

New

• !1820 Add ‘tap-spreadsheets-anywhere’ as an extractor

Changes

• #2309 Pretty print meltano schedule list --format=json output
• #2312 Don’t unnecessarily run discovery and generate catalog when running meltano invoke <extractor> --help, making it less likely to fail

Today, we are excited to release Meltano version 1.45.0, which (among other things) adds a new Loaders tab to Meltano UI that lets you easily manage and configure your project’s loaders, as an alternative to using the meltano discover, meltano add, and meltano config CLI commands:

Special thanks go out to Dmitry Stadnik of Applied Labs for contributing this functionality!

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.44.0 on August 11:

New

• #2071 Add new “Loaders” page to UI
• #2222 Add OAuth credentials settings to tap-google-analytics

Changes

• #2197 Change target-csv to use singer-io/target-csv instead of the Meltano fork

Fixes

• #2268 Fix bug causing custom plugins not to show up in meltano discover and have “Unknown” label in UI

Today, we are excited to release Meltano version 1.44.0, which (among other things) fixes a bug that caused meltano elt to fail when an extractor (Singer tap) emits a RECORD message larger than 64KB in size. The message size limit has now been raised to 1MB.

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.43.0 on August 4:

Fixes

• #2219 Don’t fail on large (record) messages output by extractors (Singer taps) by increasing subprocess output buffer size from 64KB to 1MB.
• #2215 Have meltano invoke <plugin> --help pass --help flag to plugin, instead of showing meltano invoke help message

Today, we are excited to release Meltano version 1.43.0, which (among other things) improves the output of meltano elt in three significant ways:

1. #2116 Prefix meltano elt output with meltano, tap-foo, target-bar and dbt labels as appropriate:

2. !1778 Clearly print reason that tap, target or dbt failed by repeating last output line:

3. #2214 Log Singer messages output by tap and target when meltano elt is run with --log-level flag (or cli.log_level setting) set to debug:

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.42.0 on July 29:

New

• #2116 Prefix meltano elt output with meltano, tap-foo, target-bar and dbt labels as appropriate
• !1778 Clearly print reason that tap, target or dbt failed by repeating last output line
• #2214 Log Singer messages output by tap and target when meltano elt is run with --log-level flag (or cli.log_level setting) set to debug

Changes

• !1778 Change stored error message when job was interrupted by KeyboardInterrupt from ‘KeyboardInterrupt()’ to ‘The process was interrupted’
• !1778 Disable noisy SettingsService logging when cli.log_level setting (or --log-level flag) is set to debug

Fixes

• #2212 Don’t show extractor extras _select, _metadata, and _schema as required in UI configuration form
• #2213 Provide extra context when meltano invoke airflow fails because of airflow initdb failing
• !1778 Fail gracefully when meltano install fails to install plugin(s)

Today, we are excited to release Meltano version 1.42.0, which (among other things) makes meltano elt and its state management more resilient to failure and interruption:

• #2147 Continuously persist state messages output by loader (forwarded from extractor) so that subsequent runs can pick up where a failed run left off
• #2180 Mark pipeline job as failed when process is interrupted (SIGINT) or terminated (SIGTERM).
• #2166 Don’t fail on large extractor state messages by increasing loader output buffer size from 64 to 128KB

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.41.1 on July 24:

New

• #2162 Print link to plugin documentation in meltano add <plugin> and meltano config <plugin> list output

Changes

• #2200 Consistently handle CLI errors
• #2147 Continuously persist state messages output by loader (forwarded from extractor) so that subsequent runs can pick up where a failed run left off
• #2198 Don’t touch project files that may be readonly when installing transform or dashboard plugins.

Fixes

• #2199 Fix meltano discover only listing custom plugins, not known (discovered) ones
• #2166 Don’t fail on large extractor state messages by increasing loader output buffer size from 64 to 128KB
• #2180 Mark pipeline job as failed when process is interrupted (SIGINT) or terminated (SIGTERM).