Now Available: Meltano v1.66.0

Today, we are excited to release Meltano version 1.66.0, which (among other things) prevents pipelines from getting stuck in the “running” state forever when their meltano elt process is killed unceremoniously by the operating system or some other mechanism.

This is realized by automatically detecting stale runs in the system database and marking them as “failed” before meltano elt runs a new pipeline with the same Job ID, and before meltano schedule list lists the scheduled pipelines and their status (which meltano invoke airflow scheduler does periodically).

As long as a pipeline is running, meltano elt now records a heartbeat timestamp on the pipeline run row in the system database every second. A run is considered stale when 5 minutes have elapsed since the last recorded heartbeat. Older runs without a heartbeat are considered stale if they are still in the “running” state 24 hours after starting.

Excited to try it out?

To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.

What else is new?

The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.65.0 on January 12:

New

• #2483 Every second, meltano elt records a heartbeat timestamp on the pipeline run row in the system database as long as the pipeline is running.
• #2483 Before running the new pipeline, meltano elt automatically marks runs with the same Job ID that have become stale as failed. A run is considered stale when 5 minutes have elapsed since the last recorded heartbeat. Older runs without a heartbeat are considered stale if they are still in the running state 24 hours after starting.
• #2483 meltano schedule list (which is run periodically by meltano invoke airflow scheduler) automatically marks any stale run as failed.
• #2502 Add User-Agent header with Meltano version to request for remote discovery.yml manifest (typically https://www.meltano.com/discovery.yml)
• #2503 Include project ID in X-Project-ID header and project_id query param in request for remote discovery.yml manifest when send_anonymous_usage_stats setting is enabled.