Today, we are excited to release Meltano version 1.64.0, which fixes runaway memory consumption when an extractor outputs records at a much higher rate than the loader can process them, by enabling flow control with a 64KB buffer size limit.
As a result of this bug,
meltano elt pipelines composed of fast extractors and slow loaders would sometimes be terminated by the operating system before completing, to prevent the system from running out of memory entirely.
Shortly after v1.64.0 was released, Yordan Ivanov reported a new critical bug introduced by this “fix”: when the extractor finishes before the loader, not all messages (records) would actually make it to the loader, but
meltano elt would finish successfully anyway. This has been fixed in Meltano version 1.64.1, released on January 8.
Excited to try it out?
To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run
meltano upgrade. This will upgrade the
meltano package and apply any necessary changes to your project.
What else is new?
- #2478 Fix runaway memory usage (and possible out-of-memory error) when extractor outputs messages at higher rate than loader can process them, by enabling flow control with a 64KB buffer size limit