Today, we are excited to release Meltano version 1.64.0, which fixes runaway memory consumption when an extractor outputs records at a much higher rate than the loader can process them, by enabling flow control with a 64KB buffer size limit.
As a result of this bug, meltano elt pipelines composed of fast extractors and slow loaders would sometimes be terminated by the operating system before completing, to prevent the system from running out of memory entirely.
Making the buffer size (and the related Singer message length limit) configurable is being tracked in a separate issue that is also being worked on this week.
Shortly after v1.64.0 was released, Yordan Ivanov reported a new critical bug introduced by this “fix”: when the extractor finishes before the loader, not all messages (records) would actually make it to the loader, but meltano elt would finish successfully anyway. This has been fixed in Meltano version 1.64.1, released on January 8.
Excited to try it out?
To upgrade Meltano and your Meltano project to the latest version, navigate to your project directory, activate the appropriate virtual environment, and run meltano upgrade. This will upgrade the meltano package and apply any necessary changes to your project.
What else is new?
The list below (copied from the changelog) covers all of the changes made to Meltano since the release of v1.63.0 on January 4:
Fixes
- #2478 Fix runaway memory usage (and possible out-of-memory error) when extractor outputs messages at higher rate than loader can process them, by enabling flow control with a 64KB buffer size limit