Now Available: Meltano v1.47.0

Today, we are excited to release Meltano version 1.47.0, which adds support for Bing Ads (thanks, VividFront!), prints docs and repo URLs when adding plugins, lets you specify a full schema for taps that can’t discover theirs, automatically uppercases target-snowflake‘s schema setting (thanks, Nevin Morgan!), and fixes a bug with embedded reports in the UI (thanks, Allan Whatmough!).

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.46.0 on August 27:

New

  • #2210 Print documentation and repository URLs when adding a new plugin to the project
  • #2277 Add tap-bing-ads as a known extractor
  • #2328 Add new upcase_string setting value_processor that will convert string values to uppercase

Changes

  • #2216 Add stream properties defined in an extractor’s schema extra to catalog if they do not exist in the discovered stream schema yet

Fixes

  • #2338 Once again change target-csv to use singer-io/target-csv instead of the Meltano fork
  • #2235 Make embed links accessible when not authenticated
  • #2328 Always convert target-snowflake schema setting value to uppercase before passing it to plugin

Building Meltano in Public: 6-weekly recap

Earlier this week, it was my turn to host a GitLab Group Conversation (a publicly live streamed Q&A on the GitLab Unfiltered YouTube channel) on Meltano.

Just like last time, I used the opportunity to provide a recap of the progress that’s been made since the last update 6 weeks ago, and to share our priorities for the upcoming weeks.

If you’re curious, check out the presentation on Google Slides and the Q&A on YouTube. The presentation content is also reproduced below, as is an embedded video of the Q&A!

Group Conversation Presentation

6 releases since the last GC (2020-07-21)

  1. V1.41.1 lets you easily override Singer stream schema descriptions for specific attributes (e.g. {“type”: [“string”, “null”], “format”: “date-time”}) using a new schema extractor extra.
  2. V1.42.0 makes meltano elt and its state management more resilient to failure and interruption.
  3. V1.43.0 improves the output of meltano elt in three significant ways.
  4. V1.44.0 fixes a bug that caused meltano elt to fail when an extractor (Singer tap) emits a RECORD message larger than 64KB in size.
  5. V1.45.0 adds a new Loaders tab to Meltano UI that lets you easily manage and configure your project’s loaders, as an alternative to using the meltano discover, meltano add, and meltano config CLI commands.
  6. V1.46.0 brings out-of-the-box support for extractor tap-spreadsheets-anywhere (docs), which lets you pull data from CSV and Excel files stored locally or in the cloud (S3, SFTP, HTTP, etc).

A lot of work was also done on reorganizing the documentation, which brought us: 

16 community contributions since the last GC (2020-07-21)

Done

  1. List installed and available extractors separately on Extractors page by Dmitry Stadnik (RFA)
  2. Update Docker file for faster builds by Paul Blankley
  3. Add Jobs resource to tap-gitlab by Charles Julian Knight
  4. Add Loaders page to UI by Dmitry Stadnik (RFA)
  5. Properly handle columns with reserved names in target-snowflake by Eric Simmerman
  6. Add oauth_credentials to tap-google-analytics settings in discovery.yml by Zachary Wynegar
  7. Use `singer-io/target-csv` instead of our own fork by Zachary Wynegar
  8. Add tap-spreadsheets-anywhere by Eric Simmerman

In review

  1. Create a JSON schema for `meltano.yml` and publish it on schemastore.org by Zachary Wynegar
  2. Improve docker-compose file template by Nevin Morgan

In development

  1. Allow system database schema to be overridden when PostgreSQL is used by Charles Julian Knight
  2. Upgrade `facebook_business` to 8.0.0 in tap-facebook by Zachary Wynegar
  3. Add support for `<ENV>_FILE` config env vars to support Docker Secrets by Zachary Wynegar
  4. Add Google Cloud Composer tutorial by Paul Tiplady
  5. Override auth check when using a shared embed link by Allan Whatmough
  6. Allow extractor entities to be selected in UI by Dmitry Stadnik (RFA)

Upcoming milestone priorities

Milestone issue board

Features

Bugs

Documentation

Epics for upcoming priorities

(Exact order to be determined)

Slack activity over the last 30 days

Join us on Slack!

Group Conversation Q&A

Now Available: Meltano v1.46.0

Today, we are excited to release Meltano version 1.46.0, which (among other things) brings out-of-the-box support for extractor tap-spreadsheets-anywhere (docs), which lets you pull data from CSV and Excel files stored locally or in the cloud (S3, SFTP, HTTP, etc):

meltano add extractor tap-spreadsheets-anywhere

To learn more about tap-spreadsheets-anywhere, read the docs and the README.

Special thanks go out to Eric Simmerman for building this tap and contributing support to Meltano!

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.45.0 on August 17:

New

  • !1820 Add ‘tap-spreadsheets-anywhere’ as an extractor

Changes

  • #2309 Pretty print meltano schedule list --format=json output
  • #2312 Don’t unnecessarily run discovery and generate catalog when running meltano invoke <extractor> --help, making it less likely to fail

Now Available: Meltano v1.45.0

Today, we are excited to release Meltano version 1.45.0, which (among other things) adds a new Loaders tab to Meltano UI that lets you easily manage and configure your project’s loaders, as an alternative to using the meltano discover, meltano add, and meltano config CLI commands:

Special thanks go out to Dmitry Stadnik of RFA for contributing this functionality!

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.44.0 on August 11:

New

  • #2071 Add new “Loaders” page to UI
  • #2222 Add OAuth credentials settings to tap-google-analytics

Changes

  • #2197 Change target-csv to use singer-io/target-csv instead of the Meltano fork

Fixes

  • #2268 Fix bug causing custom plugins not to show up in meltano discover and have “Unknown” label in UI

Now Available: Meltano v1.44.0

Today, we are excited to release Meltano version 1.44.0, which (among other things) fixes a bug that caused meltano elt to fail when an extractor (Singer tap) emits a RECORD message larger than 64KB in size. The message size limit has now been raised to 1MB.

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.43.0 on August 4:

Fixes

  • #2219 Don’t fail on large (record) messages output by extractors (Singer taps) by increasing subprocess output buffer size from 64KB to 1MB.
  • #2215 Have meltano invoke <plugin> --help pass --help flag to plugin, instead of showing meltano invoke help message

Now Available: Meltano v1.43.0

Today, we are excited to release Meltano version 1.43.0, which (among other things) improves the output of meltano elt in three significant ways:

  1. #2116 Prefix meltano elt output with meltano, tap-foo, target-bar and dbt labels as appropriate:
meltano elt tap-gitlab target-postgres --transform=run
  1. !1778 Clearly print reason that tap, target or dbt failed by repeating last output line:
meltano elt tap-gitlab target-jsonl
  1. #2214 Log Singer messages output by tap and target when meltano elt is run with --log-level flag (or cli.log_level setting) set to debug:
meltano --log-level=debug elt tap-gitlab target-postgres

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.42.0 on July 29:

New

  • #2116 Prefix meltano elt output with meltano, tap-foo, target-bar and dbt labels as appropriate
  • !1778 Clearly print reason that tap, target or dbt failed by repeating last output line
  • #2214 Log Singer messages output by tap and target when meltano elt is run with --log-level flag (or cli.log_level setting) set to debug

Changes

  • !1778 Change stored error message when job was interrupted by KeyboardInterrupt from ‘KeyboardInterrupt()’ to ‘The process was interrupted’
  • !1778 Disable noisy SettingsService logging when cli.log_level setting (or --log-level flag) is set to debug

Fixes

  • #2212 Don’t show extractor extras _select, _metadata, and _schema as required in UI configuration form
  • #2213 Provide extra context when meltano invoke airflow fails because of airflow initdb failing
  • !1778 Fail gracefully when meltano install fails to install plugin(s)

Now Available: Meltano v1.42.0

Today, we are excited to release Meltano version 1.42.0, which (among other things) makes meltano elt and its state management more resilient to failure and interruption:

  • #2147 Continuously persist state messages output by loader (forwarded from extractor) so that subsequent runs can pick up where a failed run left off
  • #2180 Mark pipeline job as failed when process is interrupted (SIGINT) or terminated (SIGTERM).
  • #2166 Don’t fail on large extractor state messages by increasing loader output buffer size from 64 to 128KB

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.41.1 on July 24:


New

  • #2162 Print link to plugin documentation in meltano add <plugin> and meltano config <plugin> list output

Changes

  • #2200 Consistently handle CLI errors
  • #2147 Continuously persist state messages output by loader (forwarded from extractor) so that subsequent runs can pick up where a failed run left off
  • #2198 Don’t touch project files that may be readonly when installing transform or dashboard plugins.

Fixes

  • #2199 Fix meltano discover only listing custom plugins, not known (discovered) ones
  • #2166 Don’t fail on large extractor state messages by increasing loader output buffer size from 64 to 128KB
  • #2180 Mark pipeline job as failed when process is interrupted (SIGINT) or terminated (SIGTERM).

Now Available: Meltano v1.41.1

Today, we are excited to release Meltano version 1.41.1, which (among other things) lets you easily override Singer stream schema descriptions for specific attributes (e.g. {"type": ["string", "null"], "format": "date-time"}) using a new schema extractor extra:

# In `meltano.yml`
extractors:
- name: tap-postgres
  pip_url: tap-postgres
  schema:
    some_table:
      created_at:
        type: ["string", "null"]
        format: date-time

This plugin extra can also be thought of and interacted with as a setting named _schema, with _schema.<entity>.<attribute> and _schema.<entity>.<attribute>.<key> nested properties:

# On the command line
meltano config <plugin> set _schema <entity> <attribute> <schema description>
meltano config <plugin> set _schema <entity> <attribute> <key> <value>

export <NAMESPACE>__SCHEMA='{"<entity>": {"<attribute>": {"<key>": "<value>"}}}'

# Once schema descriptions have been set in `meltano.yml`, 
# environment variables can be used to override specific
# nested properties:
export <NAMESPACE>__SCHEMA_<ENTITY>_<ATTRIBUTE>_<KEY>=<value>

# For example:
meltano config tap-postgres set _metadata some_table created_at type '["string", "null"]'
meltano config tap-postgres set _metadata some_table created_at format date-time

export TAP_POSTGRES__SCHEMA_SOME_TABLE_CREATED_AT_FORMAT=date

Whenever an extractor is run using meltano elt or meltano invoke, Meltano will generate the desired catalog on the fly by running the tap in discovery mode and applying the selection, metadata, and schema rules to the resulting catalog file before passing it to the tap in sync mode.

Check out the brand-new documentation for more about plugin extras in general and the select, metadata, and schema extractor extras in particular.


A special thanks goes out to Dmitry Stadnik of RFA, who contributed an improvement to Meltano UI: List installed and available extractors separately on Extractors page.

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.41.0 on July 20:

New

  • #2196 Pretty print and apply appropriate nesting to meltano config output
  • #2003 Let extractor extra select be interacted with as _select setting
  • !1774 Let transform extra vars be interacted with as _vars setting
  • !1774 Add support for kind: object settings, which can gather nested values from across setting stores
  • !1774 Add support for aliases: [...] on setting definitions
  • !1774 Add support for value_processor: 'nest_object' on setting definitions
  • #2145 Let discovered catalog schema be modified using schema rules stored in extractor schema extra (aka _schema setting)

Changes

  • #2070 List installed and available extractors separately on Extractors page
  • !1774 Replace update.* file bundle settings with update extra (aka _update setting)
  • !1774 Replace metadata.* extractor settings with metadata extra (aka _metadata setting)
  • #1764 Interpret meltano config <plugin> set value as JSON to allow non-string values to be set more easily

Fixes

  • !1774 Fix poor performance of setting lookups using more memoization

Building Meltano in Public: 6-weekly recap

Earlier this week, it was my turn to host a GitLab Group Conversation (a publicly live streamed Q&A on the GitLab Unfiltered YouTube channel) on Meltano. Just like last time, I used the opportunity to provide a recap of what’s happened in the community and the progress that’s been made since the last update 6 weeks ago.

If you’re curious, check out the presentation on Google Slides and the Q&A on YouTube. The presentation content is also reproduced below, as is an embedded video of the Q&A!

Group Conversation Presentation

12 releases since the last GC (2020-06-09), with 72 changelog items!

  1. V1.35.0 (10 changelog items) makes it easier to manage and version control a Meltano project’s configuration by having meltano config store values in meltano.yml by default, instead of the system database.
  2. V1.35.1 (3 changelog items) fixes 2 minor bugs and makes the output of meltano add and meltano install consistent in how they refer to plugin types and names.
  3. V1.36.0 (7 changelog items) introduces “file bundle” plugins that are used to automatically add plugin-specific files to your Meltano project when adding a plugin like the dbt transformer or Airflow orchestrator.
  4. V1.36.1 (5 changelog items) adds the ability to specify Singer tap stream and property metadata as if it were any other plugin configuration.
  5. V1.37.0 (4 changelog items) lets you instantly containerize your Meltano project using meltano add files docker.
  6. V1.38.0 (5 changelog items) helps you debug your plugin configuration by having meltano config <plugin> list show the current values of settings and where they come from, alongside the setting names and their environment variables.
  7. V1.38.1 (8 changelog items) contains a significant 61-file refactor of how project settings and environment variables are managed internally.
  8. V1.39.0 (6 changelog items) enables more streamlined and consistent configuration of all Meltano and Meltano UI settings using meltano config and the same highly flexible layered configuration model already used to manage plugin settings.
  9. V1.39.1 (9 changelog items) adds various new settings that make it easier to customize Meltano UI and configure it for production.
  10. V1.40.0 (1 changelog item) lets you customize how your custom extractors show up in Meltano UI using a new logo_url property and existing label and description properties. 
  11. V1.40.1 (9 changelog items) makes meltano config and the UI automatically store configuration in the most appropriate location, taking into account whether the project is deployed as read-only, and whether the setting is sensitive or environment-specific. 
  12. V1.41.0 (6 changelog items) adds a --full-refresh flag to meltano elt to perform a full refresh, ignoring any state left behind by previous runs with the same job ID.

Slack activity over the last 30 days

Join us on Slack!

Slack activity since May 1

I’m not sure why we’re seeing a downward trend in weekly active members over the last week, but if that hasn’t turned around by the next GC, I promise I’ll have more to say about it then!

Other things that have happened since the last GC (2020-06-09)

Current milestone

This week, I’ll be working on these issues filed or requested by the community:

RFA is expected to work on:

Group Conversation Q&A