Building Meltano in Public: 6-weekly recap

Last week, it was once again my turn to host a GitLab Group Conversation (a publicly live streamed Q&A on the GitLab Unfiltered YouTube channel) on Meltano!

I used the opportunity to share a recap of:

If you’re curious, check out the presentation on Google Slides and the Q&A on YouTube. The presentation content is also reproduced below, as is an embedded video of the Q&A!

Group Conversation Presentation

5 releases since the last GC (2020-10-07)

  1. V1.54.0 ensures arbitrary environment variables defined in .env are passed to invoked plugins, recreates plugin virtual environments when running meltano install to make sure the latest versions of dependencies are installed, and bumps the Airflow version to 1.10.12.
  2. V1.55.0 adds out-of-the-box support for the datamill-co and transferwise (aka PipelineWise) variants of target-snowflake. The datamill-co variant is recommended for new users and is now the default, but the original meltano variant is still supported. 
  3. V1.56.0 adds out-of-the-box support for the BigQuery destination using the target-bigquery loader, as well as official support for Python 3.8 alongside 3.6 and 3.7! 
  4. V1.57.0 adds out-of-the-box support for the PostgreSQL and MySQL/MariaDB data sources using the transferwise (aka PipelineWise) variants of tap-postgres and tap-mysql.
  5. V1.58.0 lets you disable autoscrolling in the pipeline log UI, fixes incorrectly disabled Test and Save buttons in the plugin configuration UI, and fixes logging in using Google Chrome when the UI is not served over HTTPS.

(Yesterday, we also released V1.59.0!)

18 recent contributions by 9 community members

Done

  1. Rebuild virtual environments when running meltano install by Niall Woodward (Tails.com)
  2. Correct docsDir configuration by Robert Speicher (GitLab)
  3. Correct README `dev:docs` command to `serve:docs` by Robert Speicher (GitLab)
  4. added OSerror catch for file system incompatibility by Nil
  5. files-airflow: Added file exists check for meltano symlink, changed logging to logger by Nil
  6. tap-gitlab:  Allow start_date to override state timestamp by Tom Elliff (CloudNC)
  7. tap-gitlab:  docs: Default for GitLab.com is the v4 API by Zeger-Jan van de Weg (GitLab)
  8. Resolve “Add `target-bigquery` loader” by Nil
  9. Officially support py38 with ci testing by Niall Woodward (Tails.com)
  10. Resolve “Add `singer-io` variant of `tap-recharge`” by Nil
  11. Refactor discovery.yaml in lexicographically ascending order by Nil
  12. Let user disable autoscrolling in UI job log modal by Niall Woodward (Tails.com)
  13. Added check for date/datetime and convert to string for meltano config <plugin> by Niltap-facebook: Bump facebook business to `8.0.5` by Paul Blankley (Zenlytic)

In development

  1. Enable `pool_pre_ping` for the project’s DB engine by Suyash Behera (Goldman Sachs)
  2. Update the ELT runner to explicitly log any error messages by Allan Whatmough (Run with AI)
  3. Search and replace “entities” >> “streams”, “attributes” >> “properties” by AJ Steers (Slalom)Add pipx-based install instructions by AJ Steers (Slalom)

Recent weekly Slack activity

In the graph on the below starting June 10, you can see how Slack activity has been growing steadily but dropped significantly during the week project lead Douwe Maan was out on vacation, with a low point on October 20. Fortunately, it recovered just as quickly once he returned!

Join us on Slack!

Other exciting recent and ongoing developments

This week’s priorities

Milestone issue board

Upcoming priorities

Milestones issue board

Epics:

Group Conversation Q&A

Now Available: Meltano v1.59.0

Today, we are excited to release Meltano version 1.59.0, which (among other things) makes sure that all meltano elt errors properly make it into the log file and UI (thanks Allan Whatmough!) and that meltano select --list prints entities and attributes in alphabetical order (thanks Nil!).

Additionally, the undocumented and experimental plugin configuration profiles feature has now been removed to make room for a more flexible solution coming in the next few weeks: plugin inheritance.

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.58.0 on November 12:

Changes

  • #2450 Remove undocumented plugin configuration profile functionality

Fixes

  • #2451 Correctly show CLI error messages in log output
  • #2453 Correctly pass value of tap-facebook‘s insights_buffer_days setting to tap as integer instead of boolean
  • #2387 Order of attributes in meltano select --list --all is set to alphabetical order.
  • #2458 Adds missing mysql-logo.png

Tutorial: Build a Slack Dashboard Using Meltano and Superset

This is a guest blog post by Srini Kadamati, a Senior Developer Advocate at Preset.io. It was originally published on the Preset.io blog on September 23, 2020.


Slack is one of the most popular community tools out there. Even though it was originally conceived as a tool for teams to communicate and coordinate, Slack embraced a free tier that lets community organizers create workspaces with unlimited members and unlimited channels.

Slack Free Tier
Slack pricing page

This lead to the meteoric rise of thousands of Slack communities across the world. You can find Slack communities for many of your professional interests, personal interests, social causes, and many more. In fact, the website Slofile will even help you discover new Slack communities to join.

Slack and Open Source

Here at Preset, we’re the experts of Apache Superset (incubating) and we invest a lot in the Superset Community. The Superset community is spread across many channels, but a lot of discussions and conversations happen in the community Slack.

As an organization focused on enabling insights through data exploration, visualizations, and dashboards, we’re always interested in finding productive ways to use data to help us understand the growth of the Superset project and the health of the Superset community.

Wouldn’t you love to have a dashboard like this for your community?

Slack dashboard

In this blog post, we’ll explore how we can leverage a few different open source tools to build an awesome dashboard for a Slack community:

  • Meltano for ELT (or extract-load-transform)
  • Postgres for data storage
  • Superset for creating views, visualizations, and dashboards

Let’s start by focusing on how to use Meltano to extract data from the Slack API.

Pipeline

Meltano = Singer + dbt

Meltano is an open source ELT platform for data integration that facilitates:

  • the extraction of data from 3rd party SaaS services
  • the loading of data into data stores
  • the transformation of the data

Under the hood, Meltano elegantly combines two open source projects: Singer and dbt.

Singer is open source and the community has created hundreds of data extractors, called Singer taps, to easily pull out data from 3rd party SaaS tools like Gitlab, Google Analytics, Hubspot, and Slack. Using Singer targets, Meltano can load data to destinations like Google BigQuery, Google Sheets, PostgreSQL, and Amazon S3.

Meltano packages both of these tools into a single platform and provides a command line interface and a graphical user interface to interact with everything very easily.

Once you have everything setup, you only need to run a single command to sync your data incrementally from your community Slack into a Postgres database:

meltano elt tap-slack target-postgres --job_id=daily_update

But we’re getting ahead of ourselves 😃

Slack API Access

Slack provides an API that’s accessible even for workspaces on the free tier (albeit with more limitations). Unfortunately, it’s not as simple as getting access to the API. You actually have to create a Slack app, set up permissions, and add the app to the channels you want to extract data out of.

In the following steps, we’ll walkthrough how to get the access you need.

1. If you’re not a Workspace Admin, I recommend getting permission from a Workspace Owner first for this project. Then, ask them to elevate your permissions to the Admin level.

2. Open the Slack workspace and navigate to [Workspace name] > Setting & administration > Customize [Workspace name].

Slack Customize

3. In the new browser page that opens, navigate to Menu > Configure apps.

Configure Apps

4. Select Build in the top right corner.

Build Slack App

5. Read the Terms of Service agreement and select I agree (if you agree!). Then select Create new app, give your app a memorable name, and select the workspace you want the app to have access to.

Create Slack App

6. Let’s now give this app some permissions. Under Add features and functionality, select Permissions.

Permissions

7. We recommend giving the following permissions to this app.

Recomended permissions

8. Then, let’s add the app to the workspace by clicking Install App to Workspace.

Install App

9. Once it’s installed, you’ll see a Bot User OAuth Access Token. This is what we need for the Meltano system to be able to extract data from our workspace. Before Meltano can extract useful data from the Slack workspace, we need to add our app to all of the public channels we want to pull data from.

Open each public channel in your community Slack and run the following command:

/invite @[app_name]

If your app was named meltano, you would type the following to add the app to the channel:

/invite @meltano

To wrap up this section, save your OAuth Access Token in a safe place. We’ll need this later in this tutorial.

Installing and Initializing Meltano

Meltano’s website has the full instructions on how to install Meltano and their documentation is amazing! We’ll cover the abridged version here! Meltano is a Python 3 library that can be installed using pip (or pip3 based on your setup):

1. Create a directory for your Meltano projects and cd into it.

mkdir meltano-projects
cd meltano-projects

2. Using a Python virtual environment local to this project folder is highly recommended (use python3 instead if that’s linked to your Python 3 version).

python -m venv .venv
source .venv/bin/activate

3. You now should have the meltano command available in your command line now. We’re now ready to initialize a Meltano project.

meltano init slack

You should see a flurry of output. Take a look at the files Meltano created.

Created slack
Creating project files...
Created slack/meltano.yml
Created slack/README.md
Created slack/requirements.txt
Created slack/.gitignore
Created slack/model/.gitkeep
Created slack/extract/.gitkeep
Created slack/load/.gitkeep
Created slack/transform/.gitkeep
Created slack/analyze/.gitkeep
Created slack/notebook/.gitkeep
Created slack/orchestrate/.gitkeep
Creating system database...

This is one of the reasons Meltano works really well with version control, CI/CD, and other common web development workflows.

Installing the Slack Extractor

To extract data from a 3rd party API, Meltano needs extractor libraries. These are Python libraries that target the Singer spec and can usually be installed using pip.

Meltano comes with a few extractors out of the box, but not one for extracting data from Slack.

1. First things first, open the folder that Meltano created for our project.

cd slack

2. We’ll need to install the community extractor library for Slack, called tap-slack. Adding any library or capabilities to a Meltano project is done using the meltano add command:

meltano add --custom extractor tap-slack

Meltano will launch a step-by-step wizard that lets us provide the necessary context (upto date Meltano documentation here). We recommend the following options:

  • (namespace): tap_slack
  • (pip_url): git+https://github.com/singer-io/tap-slack.git
  • (executable): tap-slack
  • (capabilities): catalog,discover,state
  • (settings): token,start_date

Here’s a summary screenshot for your convenience:

Tap Slack Settings

3. For a visual confirmation, you can inspect the meltano.yml file.

4. By default, Meltano will instruct extractors like tap-slack to extract all of the entities and attributes specified in the extractor’s catalog. You can run the following command to list all of the attributes Meltano will extract by default:

meltano select --list --all tap-slack
Meltano Schema

You can exclude specific entities and attributes if you’d like using:

meltano select tap-slack <entity> attribute

For example, the following command will instruct Meltano to not extract teams.plan:

meltano select tap-slack --exclude teams plan

5. To finish up configuration, we need to set the token and start_date parameters that we specified to Meltano during the extractor setup wizard. If you run meltano config tap-slack, you’ll notice that these values are set to null:

{
"token": null,
"start_date": null
}

Before configuring the token value, we recommend instructing Meltano to save this OAuth token in .env instead of as plain-text in meltano.yml (which is recommended practice). This can be accomplished by opening meltano.yml and editing the settings section to match the following:

plugins:
extractors:
- name: tap-slack
namespace: tap_slack
pip_url: git+https://github.com/singer-io/tap-slack.git
executable: tap-slack
capabilities:
- catalog
- discover
- state
settings:
    - name: token
      kind: password
    - name: start_date

6. Now, let’s set the token value, use the following command:

meltano config tap-slack set token xoxb-22342-oauth1-token2-etc

Recall that the OAuth token comes from the Slack App page from earlier in this post.

7. Finally, you need to set the start_date value. This represents the earliest datetime value you want for data like messages and channels synced (all users will be synced regardless of this value).

meltano config tap-slack set start_date 2020-09-01

We recommend picking a recent start date (2020-09-01 at the time of writing is the start of the current month) to test the pipeline.

Syncing Slack Data

Now that we have all of the pieces setup, we just need to configure the destination for the data extracted from Slack. To test the extraction, let’s load the data as JSON to a local folder.

To recap on Singer and Meltano vocabulary:

  • data extraction from an API: Singer taps or Meltano extractors
  • data loading to a destination: Singer targets or Meltano loaders

Meltano comes with target-jsonl out of the box, which stands for JSON lines. Let’s use this for testing our pipeline.

1. You can add target-jsonl using a single command:

meltano add loader target-jsonl

2. Create a directory called output for target-jsonl to dump to.

mkdir -p output

To test our pipeline we can run the following, beautiful command and watch as Meltano coordinates the orchestra of commands needed for this pipeline.

meltano elt tap-slack target-jsonl --job_id=blogpost

Let’s breakdown the different components of the command while we wait for the data pipeline to finish:

  • meltano elt: ELT stands for Extract-Load-Transform and this command lets us specify we want an ELT job to run
  • tap-slack: extract data from Slack, using the associated credentials and settings we specified
  • target-jsonl: load data to a local folder, as JSON files
  • --job_id=blogpost: Meltano pipelines often generate state and that state is linked to the job_id we specified. To re-run the data pipeline in an incremental fashion (only sync the changes), we can just re-run the same exact command using the same exact job_id. The Meltano library will output a lot of logs to the terminal as it’s running. The most common error we ran into was a timeout error because the Slack API couldn’t be reached.

Slack will throttle a large volume of inbound API requests (especially on the free plan), but thankfully the Meltano system was designed to gracefully back off and try again with a time delay. With some patience, you should see the delightful success logs once the data is finished extracting and finished loading (to the folder locally).

You should see lots of colorful output from Meltano as it runs through the full data pipeline.

Colorful Meltano

The data you’re looking for will live in output/:

Output Data

Next Steps

In this post, we performed a deep dive into how to configure Meltano extracting data from Slack and loading into a local folder of JSON files.

To finish building this dashboard, read parts 2 and 3 of this blog series over at the Preset blog:

Now Available: Meltano v1.58.0

Today, we are excited to release Meltano version 1.58.0, which (among other things) lets you disable autoscrolling in the pipeline log UI, fixes incorrectly disabled Test and Save buttons in the plugin configuration UI, and fixes logging in using Google Chrome when the UI is not served over HTTPS.

To resolve this last bug, Meltano UI no longer enables the Secure flag on session cookies unconditionally and by default. If you are serving the UI over HTTPS, it is recommend that you enable the new ui.session_cookie_secure setting so that session cookies used for authentication are only sent along with secure requests.

Special thanks to go out to Niall Woodward of Tails.com for contributing the ability to disable autoscrolling in the pipeline log UI, and to Nil for contributing a bug fix that lets you use unquoted YYYY-MM-DD timestamps in your plugin configuration in meltano.yml!

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.57.0 on November 10:

New

  • #2438 Add missing replica_set, ssl, verify_mode, and include_schemas_in_destination_stream_name settings to tap-mongodb
  • #2389 Let user disable autoscrolling in UI job log modal
  • #2307 Add ui.session_cookie_secure setting to let Secure flag be enabled on session cookies when Meltano UI is served over HTTPS

Changes

  • #2307 The Secure flag is no longer enabled on Meltano UI session cookies unconditionally and by default

Fixes

  • #2396 Support unquoted YYYY-MM-DD date values in meltano.yml by converting them to ISO8601 strings before passing them to plugins
  • #2445 Fix ‘Test Connection’ and ‘Save’ buttons being disabled in UI Configuration modal despite required fields being populated
  • #2307 Fix logging into Meltano UI in Google Chrome when running without HTTPS

Now Available: Meltano v1.57.0

Today, we are excited to release Meltano version 1.57.0, which (among other things) adds out-of-the-box support for the PostgreSQL and MySQL/MariaDB data sources using the transferwise (aka PipelineWise) variants of tap-postgres and tap-mysql.

It also adds support for the datamill-co and transferwise variants of target-postgres, with the datamill-co variant now the default, while the original meltano variant remains supported.

Finally, special thanks go out to Nil for contributing support for the ReCharge data source and cleaning up discovery.yml!

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.56.0 on November 2:

New

  • #2368 Add datamill-co variant of target-postgres
  • #2368 Add transferwise variant of target-postgres
  • #2404 Add singer-io variant of tap-recharge
  • #2410 Add transferwise variant of tap-postgres
  • #2411 Add transferwise variant of tap-mysql
  • #2407 Refactor discovery.yaml to keep things in lexicographically ascending order and in correct format
  • #2435 Add new stringify setting value_post_processor that will convert values to their JSON string representation
  • #2437 Add meltano invoke --dump option with possible values config and catalog (for extractors) to dump content of plugin-specific generated file

Changes

  • #2368 Make datamill-co variant of target-postgres the default instead of meltano

Now Available: Meltano v1.56.0

Today, we are excited to release Meltano version 1.56.0, which adds out-of-the-box support for the BigQuery destination using the target-bigquery loader, as well as official support for Python 3.8 alongside 3.6 and 3.7!

Versions of the meltano/meltano Docker image that come with Python 3.7 or 3.8 instead of 3.6 are now also available, which can be identified using the -python<X.Y> image tag suffix, e.g. latest-python3.8 and v1.56.0-python3.7.

Special thanks go out to Nil for contributing support for BigQuery, and to Niall Woodward of Tails.com for contributing support for Python 3.8!

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.55.0 on October 30:

New

  • #2374 Add target-bigquery discoverable plugin definition
  • #1956 Add support for Python 3.8

Now Available: Meltano v1.55.0

Today, we are excited to release Meltano version 1.55.0, which (among other things) adds out-of-the-box support for the datamill-co and transferwise (aka PipelineWise) variants of target-snowflake. The datamill-co variant is recommended for new users and is now the default, but the original meltano variant is still supported.

Special thanks go out to Nil for contributing a bug fix to stop meltano init from failing when the underlying filesystem doesn’t support symlinks!

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.54.0 on October 8:

New

  • #2368 Add transferwise and datamill-co variants of target-snowflake

Changes

  • #2368 Make datamill-co variant of target-snowflake the default instead of meltano
  • #2380 Add target_ prefix to namespaces of discoverable loaders target-postgres, target-snowflake, and target-sqlite

Fixes

  • #2373 meltano init emits warning instead of failing when underlying filesystem doesn’t support symlinks
  • #2391 Add missing max_workers setting to tap-salesforce discoverable plugin definition
  • #2400 Constrain Airflow installation to specific set of known-to-work requirements to prevent it from breaking unexpectedly

Watch Now: Open source EL(T) with Meltano and Singer

On October 16, I had the honor to host a talk and Q&A on open source EL(T) with Meltano and Singer at a Data Nerd Herd event organized by Ternary Data‘s Joe Reis.

If you’d like to learn why GitLab is so excited about open source data integration, how Singer provides the foundation, and how Meltano completes the picture, watch the recorded talk below, or check out the slides!

Talk description

Meltano is an open source, self-hosted, CLI-first ELT platform that brings DevOps best practices such as version control, code review, and continuous integration and deployment (CI/CD) to a space historically dominated by proprietary, hosted (SaaS), UI-first solutions, that end up being poorly customizable black boxes in otherwise transparent data stacks.

At GitLab, where Meltano was founded in 2018, we think that the power of data integration should be available to all, and are on a mission to build a truly competitive alternative to existing proprietary EL(T) solutions, in terms of ease of use, reliability, and quantity and quality of supported data sources.

Meltano leverages the existing ecosystem of open source Singer taps and targets for extraction and loading, and supports dbt transformations and Airflow orchestration through a thin plugin-based integration layer. This makes Meltano the easiest way to get started with open source ELT, without imposing arbitrary limitations on those advanced users who want to take a look behind the curtain and interact with the plugins directly.

About the presenter

Douwe Maan leads the Meltano project at GitLab, which he joined in 2015 as its fifth software engineer, when GitLab’s community of open source contributors still greatly outnumbered the in-house engineering team. He became GitLab’s first Engineering Lead, and ran the team responsible for all version control and code review functionality until 2019, when he moved to Meltano to attempt to replicate GitLab’s success in commoditizing DevOps tooling in the data integration space.

Building Meltano in Public: 6-weekly recap

Earlier this week, it was my turn to host a GitLab Group Conversation (a publicly live streamed Q&A on the GitLab Unfiltered YouTube channel) on Meltano.

I used the opportunity to share a recap of:

If you’re curious, check out the presentation on Google Slides and the Q&A on YouTube. The presentation content is also reproduced below, as is an embedded video of the Q&A!

Group Conversation Presentation

Meltano has had 7 releases since the last GC (2020-09-01)

  1. V1.47.0 adds support for Bing Ads, prints docs and repo URLs when adding plugins, lets you specify a full schema for taps that can’t discover theirs, automatically uppercases target-snowflake‘s schema setting, and fixes a bug with embedded reports in the UI.
  2. V1.48.0 lets you extract a subset of selected entities using new --select and --exclude options on meltano elt, improves plugin invocation and extractor catalog discovery error messages, and changes where meltano elt logs and generated plugin config files are stored. 
  3. V1.49.0 standardizes on <PLUGIN_NAME>_<SETTING_NAME> for configuration environment variable names, makes environment variable expansion in setting values more flexible, and uses this to let you easily override your extractor’s load_schema.
  4. V1.50.0 lets you manually provide extractor catalog and state files to meltano elt using new --catalog and --state options (and catalog and state extras), as an alternative to letting the catalog be generated on the fly and letting state be looked up based on the Job ID.
  5. V1.51.0 simplifies debugging extractor catalog generation, pipeline state lookup, and pipeline-specific configuration by letting you dump the contents of meltano elt‘s generated catalog, state, and config files to STDOUT (or a file) using a new --dump option.
  6. V1.52.0 fixes a bug where meltano elt --transform=run would unexpectedly install a transform plugin, and another where meltano select would show outdated results after changing configuration.
  7. V1.53.0 lays the foundation for out-of-the-box support for different variants of extractors and loaders, like the transferwise and datamill-co variants of target-snowflake and the singer-io variant of tap-facebook.  

5 community members made 11 recent contributions

Done

  1. Compose file update and readme addition by Nevin Morgan (VividFront)
  2. Override auth check when using a shared embed link by Allan Whatmough
  3. Resolve “Add a new `upcase_string` `value_processor` and apply to `target-snowflake`’s `schema` setting” by Nevin Morgan (VividFront)
  4. Add max_active_runs=1 to prevent scheduled job overlap by Niall Woodward (Tails.com)
  5. Remove automatic plugin install and remove associated tests by Paul Blankley (Zenlytic)
  6. Remove snowflake-connector-python dependency, bump snowflake-sqlalchemy, sqlalchemy and flask-sqlalchemy by Niall Woodward (Tails.com)
  7. Stop inheriting Meltano venv when invoking Airflow by Niall Woodward (Tails.com)
  8. Upgrade `pip` and related tools to the latest version in plugin venvs by Charles Julian Knight (FIXD)

In development

  1. Add pipelinewise-tap-mysql and pipelinewise-target-snowflake to known tap and targets by Niall Woodward (Tails.com)
  2. Use pipenv for reproducible development environment by Niall Woodward (Tails.com)
  3. Bump Airflow version to 1.10.12 by Niall Woodward (Tails.com)

Recent weekly Slack activity

Join us on Slack!

Other exciting recent developments

Current milestone priorities

Milestone issue board

Epics for upcoming priorities

Group Conversation Q&A

Now Available: Meltano v1.54.0

Today, we are excited to release Meltano version 1.54.0, which (among other things) ensures arbitrary environment variables defined in .env are passed to invoked plugins, recreates plugin virtual environments when running meltano install to make sure the latest versions of dependencies are installed, and bumps the Airflow version to 1.10.12.

Special thanks go out to Niall Woodward of Tails.com for contributing these last two changes!

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.53.0 on October 6:

Changes

  • #2057 Bump Airflow version to 1.10.12
  • #2224 Delete (if present) and recreate virtual environments for all plugins when running meltano install.

Fixes

  • #2334 Omit keys for settings with null values from config.json files generated for taps and targets, to support plugins that check if a config key is present instead of checking if the value is non-null.
  • #2376 Fix meltano elt ... --transform={run,only} raising PluginMissingError when a default transform for the extractor is discoverable but not installed
  • #2377 Ensure arbitrary env vars defined in .env are passed to invoked plugins