Before you move on, make sure you have Python
1.2 or newer installed.
We welcome contributions, idea submissions, and improvements. In fact we may already have open issues labeled Accepting Merge Requests if you don't know where to start. Please see the contribution guidelines below for source code related contributions.
# Clone the Meltano repo git clone firstname.lastname@example.org:meltano/meltano.git # Change directory into the Meltano project cd meltano # Optional, but it's best to have the latest pip pip install --upgrade pip # Optional, but it's best to have the latest setuptools pip install --upgrade setuptools # Optional, but it's recommended to create a virtual environment # in order to minimize side effects from unknown environment variable python -m venv ~/virtualenvs/meltano-development # Activate your virtual environment source ~/virtualenvs/meltano-development/bin/activate # Install all the dependencies pip install -r requirements.txt # Install dev dependencies with the edit flag on to detect changes # Note: you may have to escape the .`[dev]` argument on some shells, like zsh pip install -e .[dev] # Bundle the Meltano UI into the `meltano` package make bundle
Meltano is now installed and available at
Head out to the tutorials to create your first project.
For all changes that do not involve working on Meltano UI (front-end) itself, run the following command:
# Starts both Meltano API and a production build of Meltano UI FLASK_ENV=development meltano ui
The development build of the Meltano API should be available at [http://localhost:5000/]
If you run into
/bin/sh: yarn: command not found, double check that you've got the prerequisites installed.
On a OSX, this can be solved by running
brew install yarn.
In the event you are contributing to Meltano UI and want to work with all of the frontend tooling (i.e., hot module reloading, etc.), you will need to run the following commands:
# Navigate to a Meltano project that has already been initialized # Start the Meltano API and a production build of Meltano UI that you can ignore meltano ui # Open a new terminal tab and go to your meltano directory # Install frontend infrastructure at the root directory yarn # Change directory to webapp source code cd src/webapp # Install dependencies yarn # Start local development environment yarn serve
The developement build of the Meltano UI will be available at [http://localhost:8080/]
Meltano API and CLI are both supported by a database that is managed using Alembic migrations.
Alembic is a full featured database migration working on top of SQLAlchemy.
Migrations for the system database are located inside the
To create a new migration, use the
alembic revision -m "message" command, then edit the created file with the necessary database changes. Make sure to implement both
downgrade, so the migration is reversible, as this will be used in migration tests in the future.
Each migration should be isolated from the
meltano module, so don't import any model definition inside a migration.
Meltano doesn't currently support auto-generating migration from the models definition.
To run the migrations, use
meltano upgrade inside a Meltano project.
Follow the merge request and changelog portions of this contribution section for text-based documentation contributions.
When adding supporting visuals to documentation, adhere to:
src/docs/.vuepress/public/screenshots/with a descriptive name
Meltano uses the below tools to enforce consistent code style. Explore the repo to learn of the specific rules and settings of each.
You may use
make lint to automatically lint all your code, or
make show_lint if you only want to see what needs to change.
A contributor should know the exact line-of-code to make a change based on convention
In the spirit of GitLab's "boring solutions" with the above tools and mantra, the frontend codebase is additionally sorted as follows:
imports are alphabetical and subgrouped by core -> third-party -> application with a return delineating. For example:
// core import Vue from 'vue' // third-party import lodash from 'lodash' // application import poller from '@/utils/poller' import utils from '@/utils/utils'
object properties and methods are alphabetical where
Vuex stores are the exception (
Over time we hope to automate the enforcement of the above sorting rules.
When testing your contributions you may need to ensure that your various
__pycache__ directories are removed. This helps ensure that you are running the code you expect to be running.
The below level hierarchy defines the back to front depth sorting of UI elements. Use it as a mental model to inform your UI decisions.
Within each aforementioned depth level is an interactive color hierarchy that further organizes content while communicating an order of importance for interactive elements. This interactive color hierarchy subtly influences the user's attention and nudges their focus.
After the primary, secondary, tertiary, or navigation decision is made, the button size decision is informed by:
is-smallmodifier if it is within a component that can have multiple instances
There are three fundamental markup groups in the codebase. All three are technically VueJS single-file components but each have an intended use:
Here is a technical breakdown:
<router-view-layout>as root with only two children:
<div class="container view-header">
<div class="container view-body">
<section>as root (naturally assumes a parent of
<div class="container view-body">) with one type of child:
<div class="columns">each with their needed
Searching for something to work on?
Start off by looking at our ~"Accepting Merge Request" label.
Keep in mind that this is only a suggestion: all improvements are welcome.
Meltano uses an approval workflow for all merge requests.
A contributor can ask for a review on any merge request, without this merge request being done and/or ready to merge.
Asking for a review is asking for feedback on the implementation, not approval of the merge request. It is also the perfect time to familiarize yourself with the code base. If you don’t understand why a certain code path would solve a particular problem, that should be sent as feedback: it most probably means the code is cryptic/complex or that the problem is bigger than anticipated.
Merge conflicts, failing tests and/or missing checkboxes should not be used as ground for sending back a merge request without feedback, unless specified by the reviewer.
Meltano uses changelog-cli to populate the CHANGELOG.md
changelog (new|change|fix|breaks) MESSAGE to describe your current work in progress.
$ changelog new "add an amazing feature" $ git add CHANGELOG.md
Make sure to add CHANGELOG entries to your merge requests.
Meltano releases weekly on Monday mornings, and you can find a list of upcoming changes in our CHANGELOG under "Unreleased"
|Release Date||Release Owner||Dogfooding Owner||Shadow|
|2019-09-03||Derek K.||Derek K.||--|
|2019-09-09||Ben H.||Ben H.||Douwe M.|
|2019-09-16||Yannis R.||Yannis R.||Douwe M.|
|2019-09-23||Douwe M.||Douwe M.||Danielle M.|
|2019-09-30||Danielle M.||Danielle M.|
|2019-10-07||Micael B.||Micael B.|
|2019-10-14||Derek K.||Derek K.|
Can't make your scheduled release?
If you are unable to cover an assigned week, please find someone to cover for you and submit an MR to this page with the new owner.
Meltano uses semver as its version number scheme.
Ensure you have the latest
master branch locally before continuing.
git fetch origin
Meltano uses tags to create its artifacts. Pushing a new tag to the repository will publish it as docker images and a PyPI package.
Meltano has a number of dependencies for the release toolchain that are required when performing a release. If you haven't already, please navigate to your meltano install and run the following command to install all development dependencies:
# activate your virtualenv source ./venv/bin/activate # pip install all the development dependencies pip install .[dev]
Execute the commands below:
# create and checkout the `release-next` branch from `origin/master` git checkout -B release-next origin/master # view changelog (verify changes made match changes logged) changelog view # after the changelog has been validated, tag the release make release # ensure the tag once the tag has been created, check the version we just bumped to: e.g. `0.22.0` => `0.23.0`. git describe --tags --abbrev=0 # push the tag upstream to trigger the release pipeline git push origin $(git describe --tags --abbrev=0) # push the release branch to merge the new version, then create a merge request git push origin release-next
Releasing a hotfix?
You can use
make type=patch release to force a patch release. This is useful when we need to release hotfixes.
masterand make sure to check
delete the source branch when the changes are merged.
meltano --version # command not found: meltano
pip install meltano
meltano init dogfood-workflow
Assuming there are no conflicts on the port, you can now open your Meltano instance at http://localhost:5000.
When you land on the Pipelines page, install
Private Token with your token
Fill out Postgres configurations
Run for Transforms and hit
Pipeline Schedule configurations
Catch-up Date with beginning of the month to minimize the data request
Save. Pipeline should automatically be running now
Orchestrate link on the top nav
Airflow UI should appear automatically with a DAG prepopulated
Pipeline Schedule is complete before moving onto next step
Connections in order to update settings and click
Analyze next to
Gitlab stats per user
Select "User Name" and "Total Issues Assigned" on the left
Run to the right of Query
Verify Chart appears
Save as new report with automatically populated name
Add to New Dashboard with automatically populated name
Edit functionality on existing Report
We should be good citizen about these, and use the default workflow to contribute. Most of these are on GitHub so:
We maintain a curated list of taps/targets that are expected to work out of the box with Meltano. Meltano also helps the CLI user find components via a
Get a list of extractors:
meltano discover extract tap-demo==... tap-zendesk==1.3.0 tap-marketo==... ...
Or a list of loaders
$ meltano discover load target-demo==... target-snowflake==git+https://email@example.com target-postgres==...
To build a Droplet snapshot, one should use Packer.
docker run --rm -e DIGITALOCEAN_TOKEN=<PERSONAL_TOKEN> -v $(pwd)/cloud/packer:/packer -w /packer hashicorp/packer:latest build marketplace-image.json
The snapshot should be available under
meltano-<timestamp> on DigitalOcean.
Tmuxinator is a way for you to efficiently manage multiple services when starting up Meltano.
In order to run applications, you need to run multiple sessions and have to do a lot of repetitive tasks (like sourcing your virtual environments). So we have created a way for you to start and track everything in its appropriate panes with a single command.
It's a game changer for development and it's worth the effort!
This config uses
$MELTANO_VENV to source the virtual environment from. Set it to the correct directory before running tmuxinator.
.venvin line 2 refers to your virtual environment directory in Step #1.
cd path/to/meltano MELTANO_VENV=.venv tmuxinator local