Meltano

# Settings Reference

Meltano supports a number of settings that allow you to fine tune its behavior, which are documented here. To quickly find the setting you're looking for, use the Table of Contents in the sidebar.

As described in the Configuration guide, Meltano will determine the values of these settings by first looking in the environment, then in your project's .env file, and finally in your meltano.yml project file, falling back to a default value if nothing was found.

You can use meltano config meltano list to list all available settings with their names, environment variables, and current values.

Configuration that is not environment-specific or sensitive should be stored in your meltano.yml project file and checked into version control. Sensitive values like passwords and tokens are most appropriately stored in the environment or your project's .env file.

meltano config meltano set <setting> <value>, which is used in the examples below, will automatically store configuration in meltano.yml or .env as appropriate.

# Plugin settings

For plugin settings, refer to the specific plugin's documentation (extractors, loaders), or use meltano config <plugin> list to list all available settings with their names, environment variables, and current values.

# Your Meltano project

These are settings specific to your Meltano project.

# send_anonymous_usage_stats

  • Environment variable: MELTANO_SEND_ANONYMOUS_USAGE_STATS, alias: !MELTANO_DISABLE_TRACKING (implies value false)
  • meltano init CLI option: --no_usage_stats (implies value false)
  • Default: true

By default, Meltano shares anonymous usage data with the Meltano team using Google Analytics. We use this data to learn about the size of our user base and the specific Meltano features they are (not yet) using, which helps us determine the highest impact changes we can make in each weekly release to make Meltano even more useful for you and others like you.

If enabled, Meltano will use the value of the project_id setting to uniquely identify your project in Google Analytics. This project ID is also sent along when Meltano loads the remote discovery.yml manifest from the URL identified by the discovery_url setting.

If you'd like to send the tracking data to a different Google Analytics account than the one run by the Meltano team, the Tracking IDs can be configured using the tracking_ids.* settings below.

If you'd prefer to use Meltano without sending the team this kind of data, you can disable tracking entirely using one of these methods:

  • When creating a new project, pass --no_usage_stats to meltano init
  • In an existing project, disable this send_anonymous_usage_stats setting
  • To disable tracking in all projects in one go, enable the MELTANO_DISABLE_TRACKING environment variable

When anonymous usage tracking is enabled, Meltano tracks the following events:

  • meltano init {project name}
  • meltano ui
  • meltano elt {extractor} {loader} --transform {skip, only, run}
  • meltano add {extractor, loader, transform, model, transformer, orchestrator}
  • meltano discover {all, extractors, loaders, transforms, models, transformers, orchestrators}
  • meltano install
  • meltano invoke {plugin_name} {plugin_args}
  • meltano select {extractor} {entities_filter} {attributes_filter}
  • meltano schedule add {name} {extractor} {loader} {interval}

Beyond the invocation of these commands and the identified command line arguments, Meltano does not track any other event metadata, plugin configuration, or processed data.

Finally, Meltano also tracks anonymous web metrics when browsing the Meltano UI pages.

# How to use

meltano config meltano set send_anonymous_usage_stats false

export MELTANO_SEND_ANONYMOUS_USAGE_STATS=false
export MELTANO_DISABLE_TRACKING=true

meltano init --no_usage_stats demo-project

# project_id

Used by Meltano to uniquely identify your project in Google Analytics if the send_anonymous_usage_stats setting is enabled.

# How to use

meltano config meltano set project_id <randomly-generated-token>

export MELTANO_PROJECT_ID=<randomly-generated-token>

# database_uri

  • Environment variable: MELTANO_DATABASE_URI
  • meltano * CLI option: --database-uri
  • Default: sqlite:///$MELTANO_PROJECT_ROOT/.meltano/meltano.db

Meltano stores various types of metadata in a project-specific system database, that takes the shape of a SQLite database stored inside the .meltano directory at .meltano/meltano.db by default.

You can choose to use a different system database backend or configuration using the --database-uri option of meltano subcommands, or the MELTANO_DATABASE_URI environment variable.

# How to use

meltano config meltano set database_uri postgresql://<username>:<password>@<host>:<port>/<database>

export MELTANO_DATABASE_URI=postgresql://<username>:<password>@<host>:<port>/<database>

meltano elt --database-uri=postgresql://<username>:<password>@<host>:<port>/<database> ...

# database_max_retries

This sets the maximum number of reconnection attempts in case the initial connection to the database fails because it isn't available when Meltano starts up.

Note: This affects the initial connection attempt only after which the connection is cached. Subsequent disconnections are handled by SQLALchemy

# How to use

meltano config meltano set database_max_retries 3

export MELTANO_DATABASE_MAX_RETRIES=3

# database_retry_timeout

This controls the retry interval (in seconds) in case the initial connection to the database fails because it isn't available when Meltano starts up.

Note: This affects the initial connection attempt only after which the connection is cached. Subsequent disconnections are handled by SQLALchemy

# How to use

meltano config meltano set database_retry_timeout 5

export MELTANO_DATABASE_RETRY_TIMEOUT=5

# project_readonly

Enable this setting to indicate that your Meltano project is deployed as read-only, and to block all modifications to project files through the CLI and UI in this environment.

Specifically, this prevents adding plugins or pipeline schedules to your meltano.yml project file, as well as modifying plugin configuration stored in meltano.yml or .env.

Note that meltano config <plugin> set and the UI can still be used to store configuration in the system database, but that settings that are already set in the environment or meltano.yml take precedence and cannot be overridden.

This setting differs from the ui.readonly setting in two ways:

  1. it does not block write actions in the UI that do not modify project files, like storing settings in the system database, and
  2. it also affects the CLI.

# How to use

meltano config meltano set project_readonly true

export MELTANO_PROJECT_READONLY=true

# discovery_url

Where Meltano can find the discovery.yml manifest that lists all discoverable plugins that are supported out of the box.

This manifest is used by meltano discover and meltano add, among others.

To disable downloading the remote discovery.yml manifest and only use the project-local or packaged version, set this setting to false or any other string not starting with http:// or https://.

# How to use

meltano config meltano set discovery_url https://meltano.example.com/discovery.yml
meltano config meltano set discovery_url false

export MELTANO_DISCOVERY_URL=https://meltano.example.com/discovery.yml
export MELTANO_DISCOVERY_URL=false

# meltano CLI

These settings can be used to modify the behavior of the meltano CLI.

# cli.log_level

  • Environment variable: MELTANO_CLI_LOG_LEVEL, alias: MELTANO_LOG_LEVEL
  • meltano CLI option: --log-level
  • Options: debug, info, warning, error, critical
  • Default: info

The granularity of CLI logging.

# How to use

meltano config meltano set cli log_level debug

export MELTANO_CLI_LOG_LEVEL=debug
export MELTANO_LOG_LEVEL=debug

meltano --log-level=debug ...

# meltano elt

These settings can be used to modify the behavior of meltano elt.

# elt.buffer_size

Size (in bytes) of the buffer between extractor and loader (Singer tap and target) that stores messages output by the extractor while they are waiting to be processed by the loader.

When an extractor generates messages (records) faster than the loader can process them, the buffer may fill up completely, at which point the extractor will be blocked until the loader has worked through enough messages to make half of the buffer size available again for new extractor output.

The length of a single line of extractor output is limited to half the buffer size. With a default buffer size of 10MiB, the maximum message size would therefore be 5MiB.

# How to use

meltano config meltano set elt.buffer_size 52428800 # 50MiB in bytes

export MELTANO_ELT_BUFFER_SIZE=52428800

# Meltano UI server

These settings can be used to configure the Meltano UI server.

Meltano UI feature settings and customization settings have their own sections.

# ui.bind_host

The host to bind to.

Together with the ui.bind_port setting, this setting corresponds to Gunicorn's bind setting.

# How to use

meltano config meltano set ui bind_host 127.0.0.1

export MELTANO_UI_BIND_HOST=127.0.0.1
export MELTANO_API_HOSTNAME=127.0.0.1

meltano ui --bind=127.0.0.1

# ui.bind_port

The port to bind to.

Together with the ui.bind_host setting, this setting corresponds to Gunicorn's bind setting.

# How to use

meltano config meltano set ui bind_port 80

export MELTANO_UI_BIND_PORT=80
export MELTANO_API_PORT=80
export PORT=80

meltano ui --bind-port=80

# ui.server_name

The host and port Meltano UI is available at, e.g. <host>:<port>.

The port will usually match the ui.bind_port setting, and can be omitted when the default port for HTTP (80) or HTTPS (443) is used.

Unless the ui.session_cookie_domain setting is set, this setting will be used as the session cookie domain.

If the ui.notification setting is enabled, this setting will be used to generate external URLs in notification emails.

When set, Meltano UI will only respond to requests whose hostname (Host header) matches this setting. If this is undesirable, you can set the ui.session_cookie_domain setting instead. This may be the case when Meltano UI is situated behind a load balancer performing health checks without specifying a hostname.

If the ui.authentication setting is enabled, meltano ui will print a security warning if neither this setting or the ui.session_cookie_domain setting has been set.

This setting corresponds to Flask's SERVER_NAME setting.

# How to use

meltano config meltano set ui server_name meltano.example.com

export MELTANO_UI_SERVER_NAME=meltano.example.com

meltano ui setup <server_name> can be used to generate secrets for the ui.secret_key and ui.password_salt settings, that will be stored in a your project's .env file along with the specified server_name.

meltano ui setup meltano.example.com

The domain match rule that the session cookie will be valid for.

If not set, the cookie will be valid for all subdomains of the configured ui.server_name.

If the ui.authentication setting is enabled, meltano ui will print a security warning if neither this setting or the ui.server_name setting has been set.

This setting corresponds to Flask's SESSION_COOKIE_DOMAIN setting.

# How to use

meltano config meltano set ui session_cookie_domain meltano.example.com

export MELTANO_UI_SESSION_COOKIE_DOMAIN=meltano.example.com

Enable the Secure flag on the session cookie, so that the client will only send it to the server in HTTPS requests.

The application must be served over HTTPS for this to make sense.

This setting corresponds to Flask's SESSION_COOKIE_SECURE setting.

# How to use

meltano config meltano set ui session_cookie_secure true

export MELTANO_UI_SESSION_COOKIE_SECURE=true

# ui.secret_key

A secret key that will be used for securely signing the session cookie.

If the ui.authentication setting is enabled, meltano ui will print a security warning if this setting has not been changed from the default.

This setting corresponds to Flask's SECRET_KEY setting.

# How to use

meltano config meltano set ui secret_key <randomly-generated-secret>

export MELTANO_UI_SECRET_KEY=<randomly-generated-secret>

meltano ui setup <server_name> can be used to generate secrets for the this setting and ui.password_salt, that will be stored in your project's .env file along with the specified ui.server_name.

meltano ui setup meltano.example.com

# ui.password_salt

The HMAC salt to use when hashing passwords.

If the ui.authentication setting is enabled, meltano ui will print a security warning if this setting has not been changed from the default.

This setting corresponds to Flask-Security's SECURITY_PASSWORD_SALT setting.

# How to use

meltano config meltano set ui password_salt <randomly-generated-secret>

export MELTANO_UI_PASSWORD_SALT=<randomly-generated-secret>

meltano ui setup <server_name> can be used to generate secrets for the this setting and ui.secret_key, that will be stored in your project's .env file along with the specified ui.server_name.

meltano ui setup meltano.example.com

# ui.workers

The number of worker processes meltano ui will use to handle requests.

This setting corresponds to Gunicorn's workers setting.

# How to use

meltano config meltano set ui workers 1

export MELTANO_UI_WORKERS=1
export WORKERS=1
export WEB_CONCURRENCY=1

# ui.forwarded_allow_ips

Comma-separated front-end (reverse) proxy IPs that are allowed to set secure headers to indicate HTTPS requests.

Set to * to disable checking of front-end IPs, which can be useful for setups where you don't know in advance the IP address of front-end, but you still trust the environment.

This setting corresponds to Gunicorn's forwarded_allow_ips setting.

# How to use

meltano config meltano set ui forwarded_allow_ips "*"

export MELTANO_UI_FORWARDED_ALLOW_IPS="*"
export FORWARDED_ALLOW_IPS="*"

# Meltano UI features

These settings can be used to enable certain features of Meltano UI.

Meltano UI server settings and customization settings have their own sectionss

# ui.readonly

To block all write actions in the Meltano UI, you can run it in in read-only mode.

If you're enabling the ui.authentication setting and would like to only use read-only mode for anonymous users, enable the ui.anonymous_readonly setting instead.

This setting differs from the project_readonly setting in two ways:

  1. it also blocks write actions in the UI that do not modify project files, like storing settings in the system database, and
  2. it does not affect the CLI.

# How to use

meltano config meltano set ui readonly true

export MELTANO_UI_READONLY=true
export MELTANO_READONLY=true

# ui.authentication

Use this setting to enable authentication and disallow anonymous usage of your Meltano instance.

Additionally, you will need to:

  1. Ensure your configuration is secure by setting the ui.secret_key and ui.password_salt settings, as well as ui.server_name or ui.session_cookie_domain, manually or using meltano ui setup <server_name>.

  2. Create at least one user using meltano user add.

# How to use

meltano config meltano set ui authentication true

export MELTANO_UI_AUTHENTICATION=true
export MELTANO_AUTHENTICATION=true

# ui.anonymous_readonly

When the ui.authentication setting is enabled, enabling this setting will allow anonymous users read-only access to Meltano UI. Once a user is authenticated, write actions will be available again.

This setting is especially useful when setting up a publicly available demo instance of Meltano UI for anonymous users to interact with. These users will not be able to make any changes, but admins will once they sign in.

# How to use

meltano config meltano set ui anonymous_readonly true

export MELTANO_UI_ANONYMOUS_READONLY=true

# ui.notification

Meltano can send email notifications upon certain events.

Your outgoing mail server can be configured using the mail.* settings below.

TIP

To ease the development and testing, Meltano is preconfigured to use a local MailHog instance to trap all the outgoing emails.

Use the following docker command to start it:

docker run --rm -p 1025:1025 -p 8025:8025 --name mailhog mailhog/mailhog

All emails sent by Meltano should now be available at http://localhost:8025/

# How to use

meltano config meltano set ui notification true

export MELTANO_UI_NOTIFICATION=true
export MELTANO_NOTIFICATION=true

# ui.analysis

If you are only using Meltano for data integration (and transformation), you can disable this setting to hide all functionality related to Analysis from the UI:

  • "Explore" and "Dashboards" tabs
  • "Explore" buttons in the "Pipelines" list and "Pipeline Run Log" modal

# How to use

meltano config meltano set ui analysis false

export MELTANO_UI_ANALYSIS=true

# Meltano UI customization

These settings can be used to customize certain aspects of Meltano UI.

Meltano UI server settings and feature settings have their own sections.

# ui.logo_url

Customize the logo used by Meltano UI in the navigation bar and on the sign-in page (when the ui.authentication setting is enabled).

# How to use

meltano config meltano set ui logo_url https://meltano.com/meltano-logo-with-text.svg

export MELTANO_UI_LOGO_URL=https://meltano.com/meltano-logo-with-text.svg

# Mail server

Meltano uses Flask-Mail to send emails. Take a look at the documentation to properly configure your outgoing email server.

# mail.server

meltano config meltano set mail server smtp.example.com

export MAIL_SERVER=smtp.example.com

# mail.port

meltano config meltano set mail port 25

export MAIL_PORT=25

# mail.default_sender

meltano config meltano set mail default_sender '"Example Meltano" <bot@meltano.example.com>'

export MAIL_DEFAULT_SENDER='"Example Meltano" <bot@meltano.example.com>'

# mail.use_tls

meltano config meltano set mail use_tls true

export MAIL_USE_TLS=true

# mail.username

meltano config meltano set mail username meltano

export MAIL_USERNAME=meltano

# mail.password

meltano config meltano set mail password meltano

export MAIL_PASSWORD=meltano

# mail.debug

meltano config meltano set mail debug true

export MAIL_DEBUG=true

# OAuth Service

Meltano ships with an OAuth Service to handle the OAuth flow in the Extractors' configuration.

WARNING

To run this service, you must have a registered OAuth application on the Authorization server.

Most importantly, the Redirect URI must be set properly so that the OAuth flow can be completed.

This process is specific to each Provider.

The OAuth Service is bundled within Meltano, and is automatically started with meltano ui and mounted at /-/oauth for development purposes.

As it is a Flask application, it can also be run as a standalone using:

FLASK_ENV=production FLASK_APP=meltano.oauth python -m flask run --port 9999

# oauth_service.url

Meltano provides a public hosted solution at https://oauth.svc.meltanodata.com.

The local OAuth service for development purposes is available at /-/oauth.

# How to use

meltano config meltano set oauth_service url https://oauth.svc.meltanodata.com

export MELTANO_OAUTH_SERVICE_URL=https://oauth.svc.meltanodata.com

# oauth_service.providers

To enable specific providers, use comma-separated oauth.provider names from discovery.yml. To enable all providers, use all.

# How to use

meltano config meltano set oauth_service providers facebook,google_adwords

export MELTANO_OAUTH_SERVICE_PROVIDERS=facebook,google_adwords

# oauth_service.facebook.client_id

meltano config meltano set oauth_service facebook client_id <facebook-client-id>

export OAUTH_FACEBOOK_CLIENT_ID=<facebook-client-id>

# oauth_service.facebook.client_secret

meltano config meltano set oauth_service facebook client_secret <facebook-client-secret>

export OAUTH_FACEBOOK_CLIENT_SECRET=<facebook-client-secret>

# oauth_service.google_adwords.client_id

meltano config meltano set oauth_service google_adwords client_id <google-adwords-client-id>

export OAUTH_GOOGLE_ADWORDS_CLIENT_ID=<google-adwords-client-id>

# oauth_service.google_adwords.client_secret

meltano config meltano set oauth_service google_adwords client_secret <google-adwords-client-secret>

export OAUTH_GOOGLE_ADWORDS_CLIENT_SECRET=<google-adwords-client-secret>

# OAuth Single-Sign-On

These variables are specific to Flask-OAuthlib and work with OAuth authentication with GitLab.

TIP

These settings are used for single-sign-on using an external OAuth provider.

For more information on how to get these from your GitLab application, check out the integration docs from GitLab.

# oauth.gitlab.client_id

meltano config meltano set oauth gitlab client_id <gitlab-client-id>

export OAUTH_GITLAB_CLIENT_ID=<gitlab-client-id>
export OAUTH_GITLAB_APPLICATION_ID=<gitlab-client-id>

# oauth.gitlab.client_secret

meltano config meltano set oauth gitlab client_secret <gitlab-client-secret>

export OAUTH_GITLAB_CLIENT_SECRET=<gitlab-client-secret>
export OAUTH_GITLAB_SECRET=<gitlab-client-secret>

# Analytics Tracking IDs

Google Analytics Tracking IDs to be used if the send_anonymous_usage_stats setting is enabled.

# tracking_ids.cli

  • Environment variable: MELTANO_TRACKING_IDS_CLI, alias: MELTANO_CLI_TRACKING_ID
  • Default: UA-132758957-3

Tracking ID for usage of the meltano CLI.

meltano config meltano set tracking_ids cli UA-123456789-1

export MELTANO_TRACKING_IDS_CLI=UA-123456789-1
export MELTANO_CLI_TRACKING_ID=UA-123456789-1

# tracking_ids.ui

  • Environment variable: MELTANO_TRACKING_IDS_UI, alias: MELTANO_UI_TRACKING_ID
  • Default: UA-132758957-2

Tracking ID for usage of Meltano UI.

meltano config meltano set tracking_ids ui UA-123456789-2

export MELTANO_TRACKING_IDS_UI=UA-123456789-2
export MELTANO_UI_TRACKING_ID=UA-123456789-2

# tracking_ids.ui_embed

  • Environment variable: MELTANO_TRACKING_IDS_UI_EMBED, alias: MELTANO_EMBED_TRACKING_ID
  • Default: UA-132758957-6

Tracking ID for usage of Meltano UI's Embed feature.

meltano config meltano set tracking_ids ui_embed UA-123456789-3

export MELTANO_TRACKING_IDS_UI_EMBED=UA-123456789-3
export MELTANO_EMBED_TRACKING_ID=UA-123456789-3

← Architecture →