# Plugin Management
Meltano takes a modular approach to data engineering in general and EL(T) in particular, where your project and pipelines are composed of plugins of different types, most notably extractors (Singer taps), loaders (Singer targets), transformers (dbt and dbt models), and orchestrators (currently Airflow, with Dagster in development).
# Adding a plugin to your project
If you'd like to add a discoverable plugin that's supported by Meltano out of the box, like one of the extractors and loaders listed on the Extractors and Loaders pages, refer to the "Discoverable plugins" section below.
Finally, if you'd like your new plugin to inherit from an existing plugin in your project, so that it can reuse the same package but override (parts of) its configuration, refer to the "Plugin inheritance" section.
# Discoverable plugins
meltano add <type> <name> # For example: meltano add extractor tap-gitlab meltano add loader target-postgres meltano add transformer dbt meltano add orchestrator airflow
plugins: extractors: - name: tap-gitlab variant: meltano pip_url: git+https://gitlab.com/meltano/tap-gitlab.git loaders: - name: target-postgres variant: datamill-co pip_url: singer-target-postgres transformer: - name: dbt pip_url: dbt orchestrators: - name: airflow pip_url: apache-airflow
If multiple variants of the discoverable plugin are available,
variant property is automatically set to the name of the default variant
(which is known to work well and recommended for new users),
so that your project is pinned to a specific package and its base plugin description.
variant property were omitted from the definition, Meltano would fall back on the original supported variant instead, which does not necessarily match the default.
pip install argument)
is repeated here for convenience, since you may want to update it to
point at a (custom) fork or to pin a package to a specific version.
If this property is omitted, it is inherited from the discoverable base plugin description identified by the
meltano add <type> <name> --variant <variant> # For example: meltano add loader target-postgres --variant=transferwise
As you might expect, this will be reflected in the
pip_url properties in your
meltano.yml project file:
plugins: loaders: - name: target-postgres variant: transferwise pip_url: pipelinewise-target-postgres
If you'd like to use multiple variants of the same discoverable plugin in your project at the same time, refer to "Multiple variants" under "Explicit inheritance" below.
If you've already added one variant to your project and would like to switch to another, refer to the "Switching from one variant to another" section below.
# Explicit inheritance
Alternatively, if you'd like to give the plugin a more descriptive name in your project,
you can use the
--as) option on
to explicitly inherit from the discoverable plugin instead:
meltano add <type> <name> --inherit-from <discoverable-name> # Or equivalently: meltano add <type> <discoverable-name> --as <name> # For example: meltano add extractor tap-postgres--billing --inherit-from tap-postgres meltano add extractor tap-postgres --as tap-postgres--billing
plugins: extractors: - name: tap-postgres--billing inherit_from: tap-postgres variant: transferwise pip_url: pipelinewise-tap-postgres
Note that the
pip_url properties were populated automatically by
meltano add as described above.
# Multiple variants
If you'd like to use multiple variants of the same discoverable plugin in your project at the same time, this feature will also come in handy.
Since plugins in your project need to have unique names, a discoverable plugin can only be shadowed once, but it can be inherited from multiple times, with each plugin free to choose its own variant:
meltano add loader target-snowflake --variant=transferwise --as target-snowflake--transferwise meltano add loader target-snowflake --variant=meltano --as target-snowflake--meltano
Assuming a regular (shadowing)
target-snowflake was added before using
meltano add loader target-snowflake,
the resulting inheriting plugin definitions in
meltano.yml project file will look as follows:
plugins: loaders: - name: target-snowflake variant: datamill-co pip_url: target-snowflake - name: target-snowflake--transferwise inherit_from: target-snowflake variant: transferwise pip_url: pipelinewise-target-snowflake - name: target-snowflake--meltano inherit_from: target-snowflake variant: meltano pip_url: git+https://gitlab.com/meltano/target-snowflake.git
Note that the
--variant option and
variant property are crucial here:
inherit_from can also be used to inherit from another plugin in the project,
inherit_from: target-snowflake by itself would have resulted in the new plugin inheriting from the existing
target-snowflake plugin (that uses the
datamill-co variant) instead of the discoverable plugin we're looking for.
Had there been no
target-snowflake plugin in the project yet,
inherit_from: target-snowflake would necessarily refer to the discoverable plugin,
but without a
variant the original variant would have been used rather than the default or a specific chosen one,
just like when shadowing with a
name but no
# Custom plugins
meltano add --custom <type> <name> # For example: meltano add --custom extractor tap-covid-19 meltano add --custom loader target-bigquery--custom # If you're using Docker, don't forget to mount the project directory, # and ensure that interactive mode is enabled so that Meltano can ask you # additional questions about the plugin and get your answers over STDIN: docker run --interactive -v $(pwd):/project -w /project meltano/meltano add --custom extractor tap-covid-19
Since Meltano doesn't have the base plugin description for the package in question yet,
meltano add --custom will ask you to find and provide this metadata yourself:
(Note that more context is provided in the actual command prompts.)
$ meltano add --custom extractor tap-covid-19 # Specify namespace, which will serve as the: # - identifier to find related/compatible plugins # - default database schema (`load_schema` extra) # for use by loaders that support a target schema (namespace): tap_covid_19 # Specify `pip install` argument, for example: # - PyPI package name: (pip_url): tap-covid-19 # - Git repository URL: (pip_url): git+https://github.com/singer-io/tap-covid-19.git # - local directory, in editable/development mode: (pip_url): -e extract/tap-covid-19 # Specify the package's executable name (executable): tap-covid-19 # Specify supported Singer features (executable flags) (capabilities): catalog,discover,state # Specify supported settings (`config.json` keys) (settings): api_token,user_agent,start_date
If you're adding a Singer tap or target that's listed on Singer's index of taps or targets,
simply providing the package name as
executable usually suffices.
name also typically matches the name of the package, but you are free to change it to be more descriptive.
If it's a tap or target you have developed or are developing yourself,
you'll want to set
pip_url to either a Git repository URL or local directory path.
If you add the
-e flag ahead of the local path, the package will be installed in editable mode.
To find out what
settings a tap or target supports, reference the README in the repository and/or documentation.
capabilities a tap supports (executable flags like
--state) are not described there,
you can try one of these tricks or refer directly to the source code.
plugins: extractors: - name: tap-covid-19 namespace: tap_covid_19 pip_url: tap-covid-19 executable: tap-covid-19 capabilities: - catalog - discover - state settings: - name: api_token - name: user_agent - name: start_date
constitute the plugin's base plugin description:
everything Meltano needs to know in order to be able to use the package as a plugin.
# Plugin inheritance
To add a new plugin to your project that inherits from an existing plugin,
so that it can reuse the same package but override (parts of) its configuration,
you can use the
--inherit-from option on
meltano add <type> <name> --inherit-from <existing-name> # For example: meltano add extractor tap-ga--client-foo --inherit-from tap-google-analytics meltano add extractor tap-ga--client-bar --inherit-from tap-google-analytics meltano add extractor tap-ga--client-foo--project-baz --inherit-from tap-ga--client-foo
plugins: extractors: - name: tap-google-analytics variant: meltano pip_url: git+https://gitlab.com/meltano/tap-google-analytics.git - name: tap-ga--client-foo inherit_from: tap-google-analytics - name: tap-ga--client-bar inherit_from: tap-google-analytics - name: tap-ga--client-foo--project-baz inherit_from: tap-ga--client-foo
Note that the
--inherit-from option and
inherit_from property can also be used to
explicitly inherit from a discoverable plugin.
# Installing your project's plugins
However, since this directory is included in your project's
by default, you'll need to explicitly run
before any other
meltano commands whenever you clone or pull an existing Meltano project from version control,
to install (or update) all plugins specified in your
meltano.yml project file.
To (re)install a specific plugin in your project, use
meltano install <type> <name>, e.g.
meltano install extractor tap-gitlab.
# Pinning a plugin to a specific version
When you add a plugin to your project, the plugin definition's
pip install argument)
meltano.yml project file
typically points at a PyPI package or Git repository without specifying a specific version,
to ensure that you always get the latest and (presumably) greatest.
This makes sense when a plugin is installed for the first time, but can lead to unwelcome surprises down the line, as your pipeline may break when a new version of the package is released that introduces a bug or changes its behavior in a backward-incompatible or otherwise undesireable way.
To ensure that
meltano install always installs the same version that was used when you originally got the pipeline working,
you can modify the plugin definition in your
meltano.yml project file to include a version identifier in the
The exact steps to determine the version and modify the
pip_url will depend on whether you are installing a package from PyPI or a Git repository:
# PyPI package
To pin the latest version:
Determine the latest version of the package by browsing to
https://pypi.org/project/<package>, e.g. https://pypi.org/project/tap-shopify.
At the time of writing, the latest version of
~=<version>version specifier to the
# Before: pip_url: tap-shopify # After: pip_url: tap-shopify==1.2.6 # Always install version 1.2.6 # Alternatively: pip_url: tap-shopify~=1.2.6 # Install 1.2.6 or a newer version in the 1.2.x range
# Git repository
If the plugin's
pip_url is set to a
git+http(s) URL, e.g.
pip install will look for a package in that repository.
To pin the latest version:
Determine the latest version of the package by browsing to the
https://part of the repository URL, e.g. https://gitlab.com/meltano/tap-gitlab or https://github.com/adswerve/target-bigquery, and finding the latest Git tag.
@<sha>Git ref specifier to the
# Before: pip_url: git+https://gitlab.com/meltano/tap-gitlab.git pip_url: git+https://github.com/adswerve/target-bigquery.git # After: pip_url: git+https://firstname.lastname@example.org pip_url: git+https://email@example.com # Alternatively: pip_url: git+https://gitlab.com/meltano/tap-gitlab.git@2657b89e8896face4ce320a03b8413bbc196cec9 pip_url: git+https://github.com/adswerve/target-bigquery.git@3df97b951b7eebdfa331a1ff570f1fe3487d632f
# Installing plugins from a custom Python Package Index (PyPi)
If you need to fetch packages from a custom Python Package Index (PyPi), you can set the
PIP_INDEX_URL environment variable to your custom URL before running
Dockerfile, this would look like:
ARG PIP_INDEX_URL=<your_custom_pypi_url> RUN meltano install
# Removing a plugin from your project
meltano remove <type> <name> meltano remove <type> <name> <name_two> # For example: meltano remove extractor tap-gitlab meltano remove loader target-postgres target-csv
plugins section of your
meltano.yml project file determines the plugins that make up your project, you can still manually remove a plugin from your project by deleting its entry from this file. Traces of the plugin may remain in the other locations mentioned above.
# Using a custom fork of a plugin
If you've forked a plugin's repository and made changes to it, you can update your Meltano project to use your custom fork instead of the canonical source:
plugins: extractors: - name: tap-gitlab variant: meltano pip_url: git+https://gitlab.com/meltano/tap-gitlab.git # pip_url: git+https://gitlab.com/meltano/tap-gitlab.git@ref-name
If your plugin source is stored in a private repository, you have two options:
Continue to authenticate over HTTP(S), and store your credentials in a
.netrcfile in your home directory:
machine <hostname> # e.g. gitlab.com or github.com login <username> password <personal-access-token-or-password>
Authenticate using SSH instead, and specify a
Reinstall the plugin from the new
meltano install <type> <name> # For example: meltano install extractor tap-gitlab
If your fork supports additional settings, you can set them as custom settings.
# Switching from one variant to another
If you've already added one variant to your project and would like to use another instead, you can add the new variant as a separate plugin or switch your existing plugin over to the new variant:
# Before: plugins: loaders: - name: target-postgres variant: datamill-co pip_url: singer-target-postgres # After: plugins: loaders: - name: target-postgres variant: meltano pip_url: git+https://github.com/meltano/target-postgres.git # Optional
If you don't know the new variant's
pip installargument), you can remove this property entirely so that Meltano will fall back on the default.
Reinstall the plugin from the new
meltano install <type> <name> # For example: meltano install loader target-postgres
View the current configuration using
meltano config <name> listto see if it is still valid:
meltano config <name> list # For example: meltano config target-postgres list
Because different variants often use different setting names, you will likely see some of the settings used by the old variant show up as custom settings, indicating that they are not supported by the new variant, while settings that the new variant expects show up with a
Noneor default value.
Assuming at least one setting did not carry over correctly from the old variant to the new variant, modify the plugin's configuration in your
meltano.ymlproject file to use the new setting names:
# Before: config: postgres_host: postgres.example.com postgres_port: 5432 postgres_username: my_user postgres_database: my_database # After: config: host: postgres.example.com port: 5432 user: my_user dbname: my_database
If any of the old settings are stored in places other than
meltano.yml, like a sensitive setting that may be stored in your project's
.envfile, you can unset the old setting and set the new one using
meltano config <name> unset <old_setting> meltano config <name> set <setting> <value> # For example: meltano config target-postgres unset postgres_password meltano config target-postgres set password my_password
Keep doing this until
meltano config <name> listshows a valid configuration for the new variant, without any of the old variant's settings remaining as custom settings.