Getting Started

Now that you have successfully installed Meltano and its requirements, you can create your first project.

TIP

Before you begin, you must activate the virtual environment you created in the installation process on the command line. All the commands below should be run inside this virtual environment.

Remember, to activate your virtual enviroment, you will need to run:

# Linux, OSX
$ source ~/virtualenvs/meltano/bin/activate

# Windows
$ %ALLUSERSPROFILE%\\virtualenvs\\meltano\\Scripts\\activate.bat

Creating Your First Project

Run this command in your terminal to initialize a new project:

$ meltano init PROJECT_NAME

TIP

For those new to the command line, your PROJECT_NAME should not have spaces in the name and should use dashes instead. For example, "my project" will not work; but "my-project" will.

Creating a project also creates a new directory with the name you gave it. Change to the new directory and then start Meltano with these commands:

$ cd PROJECT_NAME
$ meltano ui

Meltano is now running, so you can start adding data sources, configuring reporting databases, scheduling updates and building dashboards.

Open your Internet browser and visit http://localhost:5000 to get started.

Connecting Data Sources

When you visit http://localhost:5000, you should see:

Do this in the Meltano UI under "Pipelines" in Step 1, Extractors. http://localhost:5000/pipelines/extractors

Meltano UI with all extractors not installed initial loading screen

Selecting Entities

Data sources can contain a LOT of different entities, and you might not want Meltano to pull every data source into your dashboard. In this step, you can choose which to include by clicking the "Edit Selections" button of an installed extractor.

Do this in the Meltano UI under "Pipelines" in Step 2, Entities. http://localhost:5000/pipelines/entities

Meltano UI pipeline entities screen new install

Selecting a Reporting Database

Now that Meltano is pulling data in from your data source(s), you need to choose where and in what format you would like that data stored.

Do this in the Meltano UI under "Pipelines" in Step 3, Loaders. http://localhost:5000/pipelines/loaders

Meltano UI pipeline targets new install

Running the ELT

Now that you've selected your reporting database, you can schedule and run your ELT pipeline.

Do this in the Meltano UI under "Pipelines" in Step 4, Schedules. http://localhost:5000/pipelines/schedules

Meltano UI pipeline schedules screen create schedule

Scheduling the ELT with Orchestration

If you're using SaaS tools to manage support, sales, marketing, revenue and other business functions you know your data is constantly changing. To keep your dashboards up to date, Meltano provides Orchestration using Apache Airflow.

TIP

Right now, Airflow can not be installed from inside Meltano's UI so you need to return to your command line interface.

Run the following command:

$ meltano add orchestrator airflow
Added 'airflow' to your Meltano project.
Activated 'airflow' virtual environment.
…
Installed 'airflow'.
Added and installed orchestrators 'airflow'.

Once Airflow is installed, you can view the ELT pipeline schedule(s) created in the previous Running the ELT step via Meltano UI where a DAG gets created for each pipeline schedule.

A DAG is automatically created in Airflow and "is a collection of all the tasks you want to run, organized in a way that reflects their relationships and dependencies".

TIP

To see a list of all your scheduled DAGs within the Meltano UI under "Orchestration" you will need to kill your terminal window running the meltano ui command and then restart it. You will only need to do this the first time you install Airflow.

$ control+c
$ meltano ui

Now click "Orchestration" in the navigation bar or visit http://localhost:5000/orchestration and you will see your schedule listed within the Airflow UI.

Meltano UI first scheduled ELT in Airflow

For a deeper explanation of how to use Meltano Orchestration with Airflow, visit Meltano's Orchestration documentation.

Analyzing Your Data

Congratulations! Now that you've ingested data into Meltano, created a reporting database, and scheduled regular updates to your dataset you're ready to analyze!

There are just three steps to take:

  1. Go to http://localhost:5000/analyze
  2. Click the Install button of your desired analysis model
  3. Once installed, click the corresponding analysis model's Analyze button

Meltano UI - Models Analyze

You're Analyze page contains links for viewing corresponding analyses. Each manifests as an interactive query builder and data visualizer. Start exploring and analyzing your data and then build savable and shareable dashboards.

Begin exploring, querying, and visualizing your data using Meltano Analyze.

Meltano UI analyze example carbon emissions data explorer

After you "Run Query" you can view charts and graphs, and save interesting query results to your dashboards.

Doing More With Meltano

Learn about more Meltano recipes and functionality with Advanced Tutorials.

Last Updated: 6/24/2019, 7:41:50 PM