Welcome! If you’re ready to get started with Meltano and run an EL(T) pipeline with a data source and destination of your choosing, you’ve come to the right place!
Short on time, or just curious what the fuss is about?To get a sense of the Meltano experience in just a few minutes, watch the
“from 0 to ELT in 90 seconds” speedrun .
They can be copy-pasted right into your terminal and will take you all the way through installation, data integration (EL), data transformation (T), orchestration, and containerization with the tap-gitlab extractor and the target-jsonl and
Before you can get started with Meltano and the meltano CLI, you’ll need to install it onto your system.
To learn more about the different installation methods, refer to the Installation guide.
If you’re running Linux or macOS and have Python 3.6, 3.7, 3.8 or 3.9 installed, we recommend installing Meltano into a dedicated Python virtual environment inside the directory that will hold your Meltano projects.
- Create and navigate to a directory to hold your Meltano projects:
mkdir meltano-projects cd meltano-projects
- Create and activate a virtual environment for Meltano inside the .venv directory:
python3 -m venv .venv source .venv/bin/activate
- Install the meltano package from PyPI :
pip3 install meltano
- Optionally, verify that the meltano CLI is now available by viewing the version:
If anything’s not behaving as expected, refer to the “Local Installation” section of the Installation guide for more details.
- Pull or update the latest version of the Meltano Docker image:
docker pull meltano/meltano:latest
By default, this image comes with the oldest version of Python supported by Meltano, currently Python 3.6. If you’d like to use a newer version of Python instead, add a -python suffix to the image tag, e.g. latest-python3.8
- Optionally, verify that the meltano CLI is now available through the Docker image by viewing the version:
docker run meltano/meltano --version
Now, whenever this guide or the documentation asks you to run the meltano command, you’ll need to run it using docker run meltano/meltano as in the example above.
When running a meltano subcommand that requires access to your project (which you’ll create in the next step), you’ll also need to mount the project directory into the container and set it as the container’s working directory:
docker run -v $(pwd):/project -w /project meltano/meltano <args>
If anything’s not behaving as expected, refer to the “Installing on Docker” section of the Installation guide for more details.
Create your Meltano project
Now that you have a way of running the meltano CLI, it’s time to create a new Meltano project that (among other things) will hold the plugins that implement the various details of your ELT pipelines.
To learn more about Meltano projects, refer to the Projects concept doc.
- Navigate to the directory that you’d like to hold your Meltano projects, if you didn’t already do so earlier:
mkdir meltano-projects cd meltano-projects
- Initialize a new project in a directory of your choosing using meltano init :
meltano init <project directory name> # For example: meltano init my-meltano-project # If you're using Docker, don't forget to mount the current working directory: docker run -v $(pwd):/projects -w /projects meltano/meltano init my-meltano-project
This will create a new directory with, among other things, your meltano.yml project file:
version: 1 send_anonymous_usage_stats: true project_id: >
It doesn’t define any plugins or pipeline schedules yet, but note that the
send_anonymous_usage_stats setting is enabled by default. To disable it, change the value to false and optionally remove the project_id setting.
- Navigate to the newly created project directory:
cd <project directory> # For example: cd my-meltano-project
- Optionally, if you’d like to version control your changes, initialize a Git repository and create an initial commit:
git init git add --all git commit -m 'Initial Meltano project'
This will allow you to use git diff to easily check the impact of the meltano commands you’ll run below on your project files, most notably your meltano.yml project file.
Add an extractor to pull data from a source
Now that you have your very own Meltano project, it’s time to add some plugins to it!
The first plugin you’ll want to add is an extractor, which will be responsible for pulling data out of your data source.
To learn more about adding plugins to your project, refer to the Plugin Management guide.
- Find out if an extractor for your data source is supported out of the box by checking the Extractors list or using meltano discover :
meltano discover extractors
- Depending on the result, pick your next step:
- If an extractor is supported out of the box, add it to your project using meltano add :
meltano add extractor <plugin name> # For example: meltano add extractor tap-gitlab # If you have a preference for a non-default variant, select it using `--variant`: meltano add extractor tap-gitlab --variant=singer-io # If you're using Docker, don't forget to mount the project directory: docker run -v $(pwd):/project -w /project meltano/meltano add extractor tap-gitlab
Select entities and attributes to extract
Now that the extractor has been configured, it’ll know where and how to find your data, but not yet which specific entities and attributes (tables and columns) you’re interested in.
By default, Meltano will instruct extractors to extract all supported entities and attributes, but it’s recommended that you specify the specific entities and attributes you’d like to extract, to improve performance and save on bandwidth and storage.
To learn more about selecting entities and attributes for extraction, refer to the Data Integration (EL) guide.
If you’ve used this Singer tap before without Meltano, you may have generated a catalog file already.
If you’d like Meltano to use it instead of generating a catalog based on the entity selection rules you’ll be asked to specify below, you can skip this section and either set the catalog extractor extra or use meltano elt ‘s –catalog option when running the data integration (EL) pipeline later on in this guide.