Now that you have successfully installed Meltano and its requirements, you can create your first project.
Before you begin, you must activate the virtual environment you created in the installation process on the command line. All the commands below should be run inside this virtual environment.
Remember, to activate your virtual enviroment, you will need to run:
# Linux, OSX source ~/virtualenvs/meltano/bin/activate # Windows %ALLUSERSPROFILE%\\virtualenvs\\meltano\\Scripts\\activate.bat
Run this command in your terminal to initialize a new project:
meltano init PROJECT_NAME
For those new to the command line, your PROJECT_NAME should not have spaces in the name and should use dashes instead. For example, "my project" will not work; but "my-project" will.
Creating a project also creates a new directory with the name you gave it. Change to the new directory and then start Meltano with these commands:
cd PROJECT_NAME meltano ui
Meltano is now running, so you can start adding data sources, configuring reporting databases, scheduling updates and building dashboards.
Open your Internet browser and visit http://localhost:5000 to get started.
When you visit http://localhost:5000, you should see:
Do this in the Meltano UI under "Pipelines" in Step 1, Extractors. http://localhost:5000/pipelines/extractors
Data sources can contain a LOT of different entities, and you might not want Meltano to pull every data source into your dashboard. In this step, you can choose which to include by clicking the "Edit Selections" button of an installed extractor.
Do this in the Meltano UI under "Pipelines" in Step 2, Entities. http://localhost:5000/pipelines/entities
Now that Meltano is pulling data in from your data source(s), you need to choose where and in what format you would like that data stored.
Do this in the Meltano UI under "Pipelines" in Step 3, Loaders. http://localhost:5000/pipelines/loaders
Now that you've selected your reporting database, you can schedule and run your ELT pipeline.
Do this in the Meltano UI under "Pipelines" in Step 4, Schedules. http://localhost:5000/pipelines/schedules
If you're using SaaS tools to manage support, sales, marketing, revenue and other business functions you know your data is constantly changing. To keep your dashboards up to date, Meltano provides Orchestration using Apache Airflow.
Right now, Airflow can not be installed from inside Meltano's UI so you need to return to your command line interface.
Run the following command:
meltano add orchestrator airflow
Once Airflow is installed, you can view the ELT pipeline schedule(s) created in the previous Running the ELT step via Meltano UI where a DAG gets created for each pipeline schedule.
A DAG is automatically created in Airflow and "is a collection of all the tasks you want to run, organized in a way that reflects their relationships and dependencies".
To see a list of all your scheduled DAGs within the Meltano UI under "Orchestration" you will need to kill your terminal window running the
meltano ui command and then restart it. You will only need to do this the first time you install Airflow.
# After installing Airflow, you will need to shut down your current instance of Meltano and restart meltano ui
Now click "Orchestration" in the navigation bar or visit http://localhost:5000/orchestration and you will see your schedule listed within the Airflow UI.
For a deeper explanation of how to use Meltano Orchestration with Airflow, visit Meltano's Orchestration documentation.
If you run into issues, it is possible that you could end up with multiple instances of Airflow running at the same time. This is a known issue (#821) common if you have been working with multiple Meltano projects, or have killed Meltano UI from the command line.
To troubleshoot, run
sudo lsof -i -P | grep -i "listen" from your command line. If you see multiple instances of Python running on Port 5010, kill the first instance with
kill 12345 (using the number for your instance). Then run
meltano ui and try again.
Congratulations! Now that you've ingested data into Meltano, created a reporting database, and scheduled regular updates to your dataset you're ready to analyze!
There are just three steps to take:
You're Analyze page contains links for viewing corresponding analyses. Each manifests as an interactive query builder and data visualizer. Start exploring and analyzing your data and then build savable and shareable dashboards.
Begin exploring, querying, and visualizing your data using Meltano Analyze.
After you "Run Query" you can view charts and graphs, and save interesting query results to your dashboards.
Learn about more Meltano recipes and functionality with Advanced Tutorials.