Orchestration

Airflow

Meltano uses Airflow in order to schedule jobs. Please find below documentation on how it can be used.

Getting started

# Install airflow in your Meltano project and make it available to use via `meltano invoke` 
$ meltano add orchestrator airflow

Meltano ships out-of-the-box with a sample DAG for your current project, which is located at orchestrate/dags/meltano.py. Airflow is a full-featured orchestrator that has a lot of features that are currently outside of Meltano's scope. As we are improving this integration, Meltano will facade more of these feature to create a seamless experience using this orchestrator. Please refer to the Airflow documentation for more in-depth knowledge about Airflow.

Currently, meltano invoke gives you raw access to the underlying plugin after any configuration hooks.

# View 'meltano' dags
$ meltano invoke airflow list_dags

# Manually trigger a task to run
$ meltano invoke airflow run --raw meltano extract_load $(date -I)

# Start the airflow ui - currently starts in a separate browser
$ meltano invoke airflow webserver -D

# start the airflow scheduler, enabling background job processing
$ meltano invoke airflow scheduler -D

# trigger a dag run
$ meltano invoke airflow trigger_dag meltano

What's next?

  • Kubernetes executor support for Meltano
Last Updated: 4/5/2019, 8:31:05 PM