# Projects

At the core of the Meltano experience is your Meltano project, which represents the single source of truth regarding your ELT pipelines: how data should be integrated and transformed, how the pipelines should be orchestrated, and how the various plugins that make up your pipelines should be configured.

Since a Meltano project is just a directory on your filesystem containing text-based files, you can treat it like any other software development project and benefit from DataOps best practices such as version control, code review, and continuous integration and deployment (CI/CD).

You can initialize a new Meltano project using meltano init.

# meltano.yml project file

At a minimum, a Meltano project must contain a project file named meltano.yml, which contains your project configuration and tells Meltano that a particular directory is a Meltano project.

The only required property is version, which currently always holds the value 1.

# Configuration

At the root of meltano.yml, and usually at the top of the file, you will find project-specific configuration.

In a newly initialized project, only the send_anonymous_usage_stats setting will be set.

To learn which settings are available, refer to the Settings reference.

# Plugins

Your project's plugins, typically added to your project using meltano add, are defined under the plugins property, inside an array named after the plugin type (e.g. extractors, loaders).

Every plugin in your project needs to have:

  1. a name that's unique among plugins of the same type,
  2. a base plugin description describing the package in terms Meltano can understand, and
  3. configuration that can be defined across various layers, including the definition's config property.

A base plugin description consists of the pip_url, executable, capabilities, and settings properties, but not every plugin definition will specify these explicitly:

When inheriting a base plugin description, the plugin definition does not need to explicitly specify a pip_url (the package's pip install argument), but you may want to override the inherited value and set the property explicitly to point at a (custom) fork or to pin a package to a specific version. When a plugin is added using meltano add, the pip_url is automatically repeated in the plugin definition for convenience.

# Inheriting plugin definitions

A plugin defined with an inherit_from property inherits its base plugin description from another plugin identified by name. To find the matching plugin, other plugins in your project are considered first, followed by discoverable plugins:





 

 

plugins:
  extractors:
  - name: tap-postgres          # Shadows discoverable `tap-postgres` (see below)
  - name: tap-postgres--billing
    inherit_from: tap-postgres  # Inherits from project's `tap-postgres`
  - name: tap-bigquery--events
    inherit_from: tap-bigquery  # Inherits from discoverable `tap-bigquery`

When inheriting from another plugin in your project, its configuration is also inherited as if the values were defaults, which can then be overridden as appropriate:










 
 
 


 
 
 
 

plugins:
  extractors:
  - name: tap-google-analytics
    variant: meltano
    config:
      key_file_location: client_secrets.json
      start_date: '2020-10-01T00:00:00Z'
  - name: tap-ga--view-foo
    inherit_from: tap-google-analytics
    config:
      # `key_file_location` and `start_date` are inherited
      view_id: 123456
  - name: tap-ga--view-bar
    inherit_from: tap-google-analytics
    config:
      # `key_file_location` is inherited
      start_date: '2020-12-01T00:00:00Z' # `start_date` is overridden
      view_id: 789012

Note that the presence of a variant property causes only discoverable plugins to be considered (even if there is also a matching plugin in the project), since only these can have multiple variants:






 

 
 

plugins:
  loaders:
  - name: target-snowflake          # Shadows discoverable `target-snowflake` (see below)
    variant: datamill-co            # using variant `datamill-co`
  - name: target-snowflake--derived
    inherit_from: target-snowflake  # Inherits from project's `target-snowflake`
  - name: target-snowflake--transferwise
    inherit_from: target-snowflake  # Inherits from discoverable `target-snowflake`
    variant: transferwise           # using variant `transferwise`

To learn how to add an inheriting plugin to your project, refer to the Plugin Management guide.

# Custom plugin definitions

A plugin defined with a namespace property (but no inherit_from property) is a custom plugin that explicitly defines its base plugin description:




 
 
 
 
 
 
 
 
 
 
 

plugins:
  extractors:
  - name: tap-covid-19
    namespace: tap_covid_19
    pip_url: tap-covid-19
    executable: tap-covid-19
    capabilities:
    - catalog
    - discover
    - state
    settings:
    - name: api_token
    - name: user_agent
    - name: start_date

To learn how to add a custom plugin to your project, refer to the Plugin Management guide.

# Shadowing plugin definitions

A plugin defined without an inherit_from or namespace property implicitly inherits its base plugin description from the discoverable plugin with the same name, as a form of shadowing:



 

plugins:
  extractors:
  - name: tap-gitlab

To learn how to add a discoverable plugin to your project, refer to the Plugin Management guide.

# Variants

If multiple variants of a discoverable plugin are available, the variant property can be used to choose a specific one:




 

plugins:
  extractors:
  - name: tap-gitlab
    variant: meltano

If no variant is specified, the original variant supported by Meltano is used. Note that this is not necessarily the default variant that is recommended to new users and would be used if the plugin were newly added to the project.

# Plugin configuration

A plugin's configuration is stored under a config property. Values for plugin extras are stored among the plugin's other properties, outside of the config object:



 
 
 
 
 

extractors:
- name: tap-example
  config:
    # Configuration goes here!
    example_setting: value
  # Extras go here!
  example_extra: value

# Schedules

Your project's pipeline schedules, typically created using meltano schedule, are defined under the schedules property.

A schedule definition must have a name, extractor, loader, transform and interval:

schedules:
- name: foo-to-bar
  extractor: tap-foo
  loader: target-bar
  transform: skip
  interval: '@hourly'

Pipeline-specific configuration can be specified using environment variables in an env dictionary:







 
 
 

schedules:
- name: foo-to-bar
  extractor: tap-foo
  loader: target-bar
  transform: skip
  interval: '@hourly'
  env:
    TAP_FOO_BAR: bar
    TAP_FOO_BAZ: baz

To learn more about pipeline schedules and orchestration, refer to the Orchestration guide.

# .gitignore

A newly initialized project comes with a .gitignore file to ensure that environment-specific and potentially sensitive configuration stored inside the .meltano directory and .env file is not leaked accidentally.

All other files are recommended to be checked into the repository and shared between all users and environments that may use the project.

# .env

Optionally, your project can contain a .env file specifying environment variables used to configure Meltano and its plugins.

Typically, this file is used to store configuration that is environment-specific or sensitive, and should not be stored in meltano.yml and checked into version control.

meltano config <plugin> set will automatically store configuration in meltano.yml or .env as appropriate.

In a newly initialized project, this file will be included in .gitignore by default.

# .meltano directory

Meltano stores various files for internal use inside a .meltano directory inside your project.

These files are specific to the environment Meltano is running in, and should not be checked into version control. In a newly initialized project, this directory will be included in .gitignore by default.

While you would usually not want to modify files in this directory directly, knowing what's in there can aid in debugging:

  • .meltano/meltano.db: The default SQLite system database.
  • .meltano/logs/elt/<job_id>/<run_id>/elt.log, e.g. .meltano/logs/elt/gitlab-to-postgres/<UUID>/elt.log: meltano elt output logs for the specified pipeline run.
  • .meltano/run/bin: Symlink to the meltano executable most recently used in this project.
  • .meltano/run/elt/<job_id>/<run_id>/, e.g. .meltano/run/elt/gitlab-to-postgres/<UUID>/: Directory used by meltano elt to store pipeline-specific generated plugin config files, like an extractor's tap.config.json, tap.properties.json, and state.json.
  • .meltano/run/<plugin name>/, e.g. .meltano/run/tap-gitlab/: Directory used by meltano invoke to store generated plugin config files.
  • .meltano/<plugin type>/<plugin name>/venv/, e.g. .meltano/extractors/tap-gitlab/venv/: Python virtual environment directory that a plugin's pip package was installed into by meltano add or meltano install.

# System database

Meltano stores various types of metadata in a project-specific system database, that takes the shape of a meltano.db SQLite database stored inside the .meltano directory by default. Like all files stored in the .meltano directory, the system database is also environment-specific.

You can choose to use a different system database backend or configuration using the database_uri setting.

While you would usually not want to modify the system database directly, knowing what's in there can aid in debugging: