At the core of the Meltano experience is your Meltano project, which represents the single source of truth regarding your ELT pipelines: how data should be integrated and transformed, how the pipelines should be orchestrated, and how the various components should be configured.
Since a Meltano project is just a directory on your filesystem containing text-based files, you can treat it like any other software development project and benefit from DevOps best practices such as version control, code review, and continuous integration and deployment (CI/CD).
You can initialize a new Meltano project using
meltano.yml project file
At a minimum, a Meltano project must contain a project file named
which contains your project configuration and tells Meltano that a particular directory is a Meltano project.
The only required key is
version, which currently always holds the value
At the root of
meltano.yml, and usually at the top of the file, you will find project-specific configuration.
In a newly initialized project, only the
will be set.
To learn which settings are available, refer to the Settings reference.
At a minimum, a plugin definition must have a
name and a
pip install argument).
# Discoverable plugin references
A plugin definition without a
namespace is a reference to a discoverable plugin with the same
plugins: extractors: - name: tap-gitlab pip_url: git+https://gitlab.com/meltano/tap-gitlab.git
In the context of your project, these plugins inherit their metadata (
settings; see below) from the discoverable plugin definition.
If multiple variants of a discoverable plugin are available, the specific variant that is used in the project is identified using the
plugins: extractors: - name: tap-gitlab variant: meltano pip_url: git+https://gitlab.com/meltano/tap-gitlab.git
variant is specified, the original variant supported by Meltano is used.
Note that this is not necessarily the default variant that is recommended to new users and would be used if the plugin were newly added to the project.
# Custom plugin definitions
namespace is specified, we're dealing with a custom plugin definition instead, and additional properties
settings are available:
plugins: extractors: - name: tap-covid-19 namespace: tap_covid_19 pip_url: tap-covid-19 executable: tap-covid-19 capabilities: - catalog - discover - state settings: - name: api_token - name: user_agent - name: start_date
# Plugin configuration
extractors: - name: tap-example pip_url: tap-example config: # Configuration goes here! example_setting: value # Extras go here! example_extra: value
A schedule definition must have a
schedules: - name: foo-to-bar extractor: tap-foo loader: target-bar transform: skip interval: '@hourly'
schedules: - name: foo-to-bar extractor: tap-foo loader: target-bar transform: skip interval: '@hourly' env: TAP_FOO_BAR: bar TAP_FOO_BAZ: baz
To learn more about pipeline schedules and orchestration, refer to the Orchestration guide.
A newly initialized project comes with a
.gitignore file to ensure that
environment-specific and potentially sensitive configuration stored inside the
.meltano directory and
.env file is not leaked accidentally.
All other files are recommended to be checked into the repository and shared between all users and environments that may use the project.
Typically, this file is used to store configuration that is environment-specific or sensitive,
and should not be stored in
meltano.yml and checked into version control.
meltano config <plugin> set will automatically store configuration in
.env as appropriate.
In a newly initialized project, this file will be included in
.gitignore by default.
Meltano stores various files for internal use inside a
.meltano directory inside your project.
These files are specific to the environment Meltano is running in, and should not be checked into version control.
In a newly initialized project, this directory will be included in
.gitignore by default.
While you would usually not want to modify files in this directory directly, knowing what's in there can aid in debugging:
.meltano/meltano.db: The default SQLite system database.
meltano eltoutput logs for the specified pipeline run.
.meltano/run/bin: Symlink to the
meltanoexecutable most recently used in this project.
.meltano/run/elt/gitlab-to-postgres/<UUID>/: Directory used by
meltano eltto store pipeline-specific generated plugin config files, like an extractor's
.meltano/run/<plugin name>/, e.g.
.meltano/run/tap-gitlab/: Directory used by
meltano invoketo store generated plugin config files.
.meltano/<plugin type>/<plugin name>/venv/, e.g.
.meltano/extractors/tap-gitlab/venv/: Python virtual environment directory that a plugin's pip package was installed into by
# System database
Meltano stores various types of metadata in a project-specific system database,
that takes the shape of a
meltano.db SQLite database stored inside the
.meltano directory by default.
Like all files stored in the
.meltano directory, the system database is also environment-specific.
You can choose to use a different system database backend or configuration using the
While you would usually not want to modify the system database directly, knowing what's in there can aid in debugging:
jobtable: One row for each
meltano eltpipeline run, holding started/ended timestamps and incremental replication state.
plugin_settingstable: Plugin configuration set using
meltano config <plugin> setor the UI when the project is deployed as read-only.
usertable: Users for Meltano UI created using
meltano user add.