In order to contribute to Meltano, you will need the following:
We welcome contributions, idea submissions, and improvements. In fact we may already have open issues labeled Accepting Merge Requests if you don't know where to start. Please see the contribution guidelines below for source code related contributions.
As you contribute to Meltano, you may want to disable metrics tracking globally rather than by project. You can do this by setting the environment variable
# Add to `~/.bashrc`, `~/.zshrc`, etc, depending on the shell you use: export MELTANO_DISABLE_TRACKING=True
# Clone the Meltano repo git clone email@example.com:meltano/meltano.git # Change directory into the Meltano project cd meltano # Optional, but it's best to have the latest pip pip install --upgrade pip # Optional, but it's best to have the latest setuptools pip install --upgrade setuptools # Optional, but it's recommended to create a virtual environment # in order to minimize side effects from unknown environment variable python -m venv ~/.venv/meltano-development # Activate your virtual environment source ~/.venv/meltano-development/bin/activate # Install all the dependencies pip install -r requirements.txt # Install dev dependencies with the edit flag on to detect changes # Note: you may have to escape the .`[dev]` argument on some shells, like zsh pip install -e .[dev] # Bundle the Meltano UI into the `meltano` package make bundle
Meltano is now installed and available at
meltano, as long as you remain in your
meltano-development virtual environment!
This means that you're ready to start Meltano CLI development. For API and UI development, read on.
This section of the guide provides guidance on how to work with the Meltano API, which serves as the backend of Meltano and is built with the Python framework: Flask.
After all of your dependencies installed, we recommend opening a new window/tab in your terminal so you can run the following commands:
# Create a new Meltano project meltano init $PROJECT_NAME # Change directory into your newly created project cd $PROJECT_NAME # Start a development build of the Meltano API and a production build of Meltano UI FLASK_ENV=development meltano ui
The development build of the Meltano API & production build of the UI should now be available at http://localhost:5000/.
To debug your Python code, here is the recommended way to validate / debug your code:
# Purpose: Start a debugger # Usage: Use as a one-line import / invocation for easier cleanup import pdb; pdb.set_trace()
In the event you are contributing to Meltano UI and want to work with all of the frontend tooling (i.e., hot module reloading, etc.), you will need to run the following commands:
# Navigate to a Meltano project that has already been initialized # Start the Meltano API and a production build of Meltano UI that you can ignore meltano ui # Open a new terminal tab and go to your meltano directory # Install frontend infrastructure at the root directory yarn setup # Start local development environment yarn serve
The development build of the Meltano UI will be available at http://localhost:8080/.
If you need to change the URL of your development environment, you can do this by updating your
.env in your project directory with the following configuration:
# The URL where the web app will be located when working locally in development # since it provides the redirect after authentication. # Not require for production export MELTANO_UI_URL = ""
Watch "How taps are built" for an explanation of how Singer taps (which form the basis for Meltano extractors) work, and what goes into building new ones or verifying and modifying existing ones for various types of APIs.
Then watch "How transforms are built" for an explanation of how DBT transforms work, and what goes into building new ones for new data sources.
Whenever new functionality is introduced that changes the schema of
discovery.yml (the exact properties it supports and their types), the
src/meltano/core/bundle/discovery.yml and the
VERSION constant in
src/meltano/core/plugin_discovery_service.py must be incremented, so that older instances of Meltano don't attempt to download and parse a
discovery.yml its parser is not compatible with.
discovery.yml that only use existing properties do not constitute schema changes and do not require
version to be incremented.
When you need to make changes to
discovery.yml, these changes are not automatically detected inside of the
meltano repo during development. While there are a few ways to solve this problem, it is recommended to create a symbolic link in order ensure that changes made inside of the
meltano repo appear inside the Meltano project you initialized and are testing on.
discovery.ymlin the repo
Open your Meltano project in your terminal
Create a symbolic link by running the following command:
ln -s $YOUR_DISCOVERY_YML_PATH
Now, when you run the
ls -l command, you should see something like:
bencodezen staff 72 Nov 19 09:19 discovery.yml -> /Users/bencodezen/Projects/meltano/src/meltano/core/bundle/discovery.yml
Now, you can see your changes in
discovery.yml live in your project during development! 🎉
Each extractor (tap) and loader (target) in the
discovery.yml has a
settings property. Nested under this property are a variable amount of individual settings. In the Meltano UI these settings are parsed to generate a configuration form. To improve the UX of this form, each setting has a number of optional properties:
- settings: - name: setting_name # Required (must match the connector setting name) label: Setting Name # Optional (human friendly text display of the setting name) value: '' # Optional (Use to set a default value) placeholder: Ex. format_like_this # Optional (Use to set the input's placeholder default) kind: string # Optional (Use for a first-class input control. Default is `string`, others are `hidden`, `boolean`, `date_iso8601`, `password`, `options`, & `file`) description: Setting description # Optional (Use to provide inline context) tooltip: Here is some more info... # Optional (use to provide additional inline context) documentation: https://meltano.com/ # Optional (use to link to specific supplemental documentation) protected: true # Optional (use in combination with `value` to provide an uneditable default) env: SOME_API_KEY # Optional (use to delegate to an environment variable for overriding this setting's value)
We should be good citizen about these, and use the default workflow to contribute. Most of these are on GitHub so:
Meltano API and CLI are both supported by a database that is managed using Alembic migrations.
Alembic is a full featured database migration working on top of SQLAlchemy.
Migrations for the system database are located inside the
To create a new migration, use the
alembic revision -m "message" command, then edit the created file with the necessary database changes. Make sure to implement both
downgrade, so the migration is reversible, as this will be used in migration tests in the future.
Each migration should be isolated from the
meltano module, so don't import any model definition inside a migration.
Meltano doesn't currently support auto-generating migration from the models definition.
To run the migrations, use
meltano upgrade inside a Meltano project.
Meltano uses VuePress to manage its documentation site. Some of the benefits it provides us includes:
When adding supporting visuals to documentation, adhere to:
src/docs/.vuepress/public/screenshots/with a descriptive name
Our end-to-end tests are currently being built with Cypress.
meltano init $PROJECT_NAME
This will kick off a Cypress application that will allow you to run tests as desired by clicking each test suite (which can be found in
In the near future, all tests can flow automatically; but there are some complications that require manual triggering due to an inability to read pipeline completion.
Meltano uses the below tools to enforce consistent code style. Explore the repo to learn of the specific rules and settings of each.
You may use
make lint to automatically lint all your code, or
make show_lint if you only want to see what needs to change.
A contributor should know the exact line-of-code to make a change based on convention
In the spirit of GitLab's "boring solutions" with the above tools and mantra, the frontend codebase is additionally sorted as follows:
imports are alphabetical and subgrouped by core -> third-party -> application with a return delineating. For example:
// core import Vue from 'vue' // third-party import lodash from 'lodash' // application import poller from '@/utils/poller' import utils from '@/utils/utils'
object properties and methods are alphabetical where
Vuex stores are the exception (
Over time we hope to automate the enforcement of the above sorting rules.
When testing your contributions you may need to ensure that your various
__pycache__ directories are removed. This helps ensure that you are running the code you expect to be running.
The below level hierarchy defines the back to front depth sorting of UI elements. Use it as a mental model to inform your UI decisions.
Within each aforementioned depth level is an interactive color hierarchy that further organizes content while communicating an order of importance for interactive elements. This interactive color hierarchy subtly influences the user's attention and nudges their focus.
After the primary, secondary, tertiary, or navigation decision is made, the button size decision is informed by:
is-smallmodifier if it is within a component that can have multiple instances
There are three fundamental markup groups in the codebase. All three are technically VueJS single-file components but each have an intended use:
Here is a technical breakdown:
<router-view-layout>as root with only one child for the main view:
<div class="container view-body">
<section>as root (naturally assumes a parent of
<div class="container view-body">) with one type of child:
<div class="columns">each with their needed
Searching for something to work on?
Start off by looking at our ~"Accepting Merge Requests" label.
Keep in mind that this is only a suggestion: all improvements are welcome.
Meltano uses an approval workflow for all merge requests.
A contributor can ask for a review on any merge request, without this merge request being done and/or ready to merge.
Asking for a review is asking for feedback on the implementation, not approval of the merge request. It is also the perfect time to familiarize yourself with the code base. If you don’t understand why a certain code path would solve a particular problem, that should be sent as feedback: it most probably means the code is cryptic/complex or that the problem is bigger than anticipated.
Merge conflicts, failing tests and/or missing checkboxes should not be used as ground for sending back a merge request without feedback, unless specified by the reviewer.
Meltano uses changelog-cli to populate the CHANGELOG.md
changelog (new|change|fix|breaks) MESSAGE to describe your current work in progress.
$ changelog new "add an amazing feature" $ git add CHANGELOG.md
Make sure to add CHANGELOG entries to your merge requests.
Tmuxinator is a way for you to efficiently manage multiple services when starting up Meltano.
In order to run applications, you need to run multiple sessions and have to do a lot of repetitive tasks (like sourcing your virtual environments). So we have created a way for you to start and track everything in its appropriate panes with a single command.
It's a game changer for development and it's worth the effort!
This config uses
$MELTANO_VENV to source the virtual environment from. Set it to the correct directory before running tmuxinator.
.venvin line 2 refers to your virtual environment directory in Step #1.
cd path/to/meltano MELTANO_VENV=.venv tmuxinator local