# Projects

At the core of the Meltano experience is your Meltano project, which represents the single source of truth regarding your ELT pipelines: how data should be integrated and transformed, how the pipelines should be orchestrated, and how the various components should be configured.

Since a Meltano project is just a directory on your filesystem containing text-based files, you can treat it like any other software development project and benefit from DevOps best practices such as version control, code review, and continuous integration and deployment (CI/CD).

You can initialize a new Meltano project using meltano init.

# meltano.yml project file

At a minimum, a Meltano project must contain a project file named meltano.yml, which contains your project configuration and tells Meltano that a particular directory is a Meltano project.

The only required key is version, which currently always holds the value 1.

# Configuration

At the root of meltano.yml, and usually at the top of the file, you will find project-specific configuration.

In a newly initialized project, only the send_anonymous_usage_stats setting will be set.

To learn which settings are available, refer to the Settings reference.

# Plugins

Definitions of plugins you've added to your project using meltano add are stored under the plugins key, nested under a key named after the plugin type (e.g. extractors, loaders).

At a minimum, a plugin definition must have a name and a pip_url (its pip install argument).

# Known plugin references

A plugin definition without a namespace is a reference to a known plugin with the same name:

plugins:
  extractors:
  - name: tap-gitlab
    pip_url: git+https://gitlab.com/meltano/tap-gitlab.git

These plugins inherit their metadata (executable, capabilities, and settings; see below) from the known plugin definition.

# Custom plugins

When a namespace is specified, we're dealing with a custom plugin definition instead, and additional properties executable, capabilities, and settings are available:




 

 
 
 
 
 
 
 
 
 

plugins:
  extractors:
  - name: tap-covid-19
    namespace: tap_covid_19
    pip_url: tap-covid-19
    executable: tap-covid-19
    capabilities:
    - catalog
    - discover
    - state
    settings:
    - name: api_token
    - name: user_agent
    - name: start_date

# Plugin configuration

A plugin's configuration is stored under a config key. Values for plugin extras are stored among the plugin's other properties, outside of the config object:







 
 

extractors:
- name: tap-example
  pip_url: tap-example
  config:
    # Configuration goes here!
    example_setting: value
  # Extras go here!
  example_extra: value

# Schedules

Definitions of pipeline schedules you've created using meltano schedule are stored under the schedules key.

A schedule definition must have a name, extractor, loader, transform and interval:

schedules:
- name: foo-to-bar
  extractor: tap-foo
  loader: target-bar
  transform: skip
  interval: '@hourly'

Pipeline-specific configuration can be specified using environment variables in an env dictionary:







 
 
 

schedules:
- name: foo-to-bar
  extractor: tap-foo
  loader: target-bar
  transform: skip
  interval: '@hourly'
  env:
    TAP_FOO_BAR: bar
    TAP_FOO_BAZ: baz

To learn more about pipeline schedules and orchestration, refer to the Orchestration guide.

# .gitignore

A newly initialized project comes with a .gitignore file to ensure that environment-specific and potentially sensitive configuration stored inside the .meltano directory and .env file is not leaked accidentally.

All other files are recommended to be checked into the repository and shared between all users and environments that may use the project.

# .env

Optionally, your project can contain a .env file specifying environment variables used to configure Meltano and its plugins.

Typically, this file is used to store configuration that is environment-specific or sensitive, and should not be stored in meltano.yml and checked into version control.

meltano config <plugin> set will automatically store configuration in meltano.yml or .env as appropriate.

In a newly initialized project, this file will be included in .gitignore by default.

# .meltano directory

Meltano stores various files for internal use inside a .meltano directory inside your project.

These files are specific to the environment Meltano is running in, and should not be checked into version control. In a newly initialized project, this directory will be included in .gitignore by default.

While you would usually not want to modify files in this directory directly, knowing what's in there can aid in debugging:

  • .meltano/meltano.db: The default SQLite system database.
  • .meltano/logs/elt/<job_id>/<run_id>/elt.log, e.g. .meltano/logs/elt/gitlab-to-postgres/<UUID>/elt.log: meltano elt output logs for the specified pipeline run.
  • .meltano/run/bin: Symlink to the meltano executable most recently used in this project.
  • .meltano/run/elt/<job_id>/<run_id>/, e.g. .meltano/run/elt/gitlab-to-postgres/<UUID>/: Directory used by meltano elt to store pipeline-specific generated plugin config files, like an extractor's tap.config.json, tap.properties.json, and state.json.
  • .meltano/run/<plugin name>/, e.g. .meltano/run/tap-gitlab/: Directory used by meltano invoke to store generated plugin config files.
  • .meltano/<plugin type>/<plugin name>/venv/, e.g. .meltano/extractors/tap-gitlab/venv/: Python virtual environment directory that a plugin's pip package was installed into by meltano add or meltano install.

# System database

Meltano stores various types of metadata in a project-specific system database, that takes the shape of a meltano.db SQLite database stored inside the .meltano directory by default. Like all files stored in the .meltano directory, the system database is also environment-specific.

You can choose to use a different system database backend or configuration using the database_uri setting.

While you would usually not want to modify the system database directly, knowing what's in there can aid in debugging: