Today, we are excited to release Meltano version 1.66.0, which (among other things) prevents pipelines from getting stuck in the “running” state forever when their
meltano elt process is killed unceremoniously by the operating system or some other mechanism.
This is realized by automatically detecting stale runs in the system database and marking them as “failed” before
meltano elt runs a new pipeline with the same Job ID, and before
meltano schedule list lists the scheduled pipelines and their status (which
meltano invoke airflow scheduler does periodically).
As long as a pipeline is running,
meltano elt now records a heartbeat timestamp on the pipeline run row in the system database every second. A run is considered stale when 5 minutes have elapsed since the last recorded heartbeat. Older runs without a heartbeat are considered stale if they are still in the “running” state 24 hours after starting.
Excited to try it out?
To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run
meltano upgrade. This will upgrade the
meltano package and apply any necessary changes to your project.
What else is new?
The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.65.0 on January 12:
- #2483 Every second,
meltano eltrecords a heartbeat timestamp on the pipeline run row in the system database as long as the pipeline is running.
- #2483 Before running the new pipeline,
meltano eltautomatically marks runs with the same Job ID that have become stale as failed. A run is considered stale when 5 minutes have elapsed since the last recorded heartbeat. Older runs without a heartbeat are considered stale if they are still in the running state 24 hours after starting.
meltano schedule list(which is run periodically by
meltano invoke airflow scheduler) automatically marks any stale run as failed.
- #2502 Add
User-Agentheader with Meltano version to request for remote
discovery.ymlmanifest (typically https://www.meltano.com/discovery.yml)
- #2503 Include project ID in
project_idquery param in request for remote
send_anonymous_usage_statssetting is enabled.