If you run database replication pipelines, you know the feeling: the data is ready, the warehouse is waiting and yet your sync crawls along one record at a time while your SLA window shrinks. That bottleneck isn’t your database and it isn’t your warehouse. It’s the wire between them. Today, we’re doing something about it.
We’ve shipped Apache Arrow based BATCH support for Meltano’s database connectors and the measured numbers speak for themselves: an 8.8x throughput improvement replicating 1 million rows from MySQL to Snowflake (2,028 to 17,860 rows/sec), and a 9.6x improvement replicating 5 million rows from Postgres (30,475 to 293,483 rows/sec).
The Pain Point: Per-Record RECORD Sync Creates Back-Pressure and Caps Throughput
The Singer specification, which underlies these connectors, was built around per-record messaging. Every single row gets serialized, emitted as a RECORD message, parsed, and loaded, one at a time. For small datasets, that’s fine. For long-running, SLA-bound replication jobs moving millions of rows out of operational databases like MySQL, Postgres, or SQL Server, it creates two hard problems:
Back-pressure: The tap can only extract as fast as the target can absorb. A fast source database ends up throttled by the slowest link in the pipeline, and neither side can work at its natural pace.
Throughput ceilings: Per-record serialization overhead puts a hard cap on rows per second, no matter how much compute you throw at the problem. Pipelines that should finish in minutes stretch into hours, and SLA windows get missed.
Data engineers told us exactly what they needed: extraction that runs as fast as the source allows, ingestion that’s fast and reliable, and none of it requiring extra configuration gymnastics.
The Solution: Taps Write Arrow IPC Files, Targets Load Them Asynchronously
The new architecture decouples extraction from loading using the Singer SDK’s BATCH message specification together with Apache Arrow:
- The tap extracts at full speed, writing records to Arrow IPC batch files on local disk instead of streaming individual RECORD messages.
- The tap emits lightweight BATCH messages that simply reference those files.
- The target consumes the batch files asynchronously, at its own pace, loading data via adbc-driver-manager (or native Arrow-compatible writers) using full-refresh or MERGE-based upsert.
No more back-pressure. The source and destination each run as fast as they can, independently.
A few design decisions we think matter just as much as the speed:
- Zero configuration on the target side
Any target or mapper that sees a BATCH message pointing to an Arrow IPC file processes it automatically. Your loading step just gets faster. - One consistent opt-in pattern on the tap side
The configuration structure established for tap-mysql is reused across every connector, so enabling BATCH looks the same everywhere. - Automatic, safe fallback
Log-based (CDC) replication paths continue to use RECORD sync. The tap falls back automatically wherever Arrow BATCH isn’t applicable, so nothing breaks.
Measured Results: 8.8x on MySQL to Snowflake, 9.6x on Postgres
We ran identical datasets through identical pipelines twice, once with classic RECORD sync and once with Arrow BATCH. Here’s the side-by-side:
| Pipeline | Dataset | Rows | RECORD | BATCH | Speedup |
| tap-mysql ->mapper ->target-snowflake | mysql/wide_customers | 1M | 493.2s (2,028 rows/sec) | 56.0s (17,860 rows/sec) | 8.8x |
| tap-postgres ->target-duckdb | postgres/wide_customers | 5M | 164.1s (30,475 rows/sec) | 17.0s (293,483 rows/sec) | 9.6x |
A MySQL-to-Snowflake sync that took over eight minutes now finishes in under a minute. Five million Postgres rows land in 17 seconds instead of nearly three. And note the MySQL pipeline runs through a mapper, so the gains hold even with a transformation step in the middle.
And we’re not asking you to take our word for it. These numbers come straight from melt-bench, our orchestrator-agnostic performance test runner that executes repeatable pipeline runs on a schedule, with consistent source datasets and standardized rows/sec reporting.
Shipped Today: Arrow BATCH for MySQL and Postgres Taps, the Snowflake Target and melt-bench
Arrow/BATCH support is available today in tap-mysql (v1.5.0) and the newly released tap-postgres, paired with target-snowflake (v0.17.8-3) covering both full-refresh and MERGE-based upsert loading. tap-postgres uses adbc-driver-postgres, with no other third-party ADBC driver required, and supports BATCH for INCREMENTAL and FULL_TABLE replication, while LOG_BASED replication continues on RECORD sync. Each release ships with changelogs documenting the new capability, and driver installation steps for adbc-driver-manager are documented per database. Alongside the connectors, the melt-bench performance test runner delivers automated RECORD vs BATCH comparisons for both MySQL and Postgres pipelines, so the throughput gains are measured, repeatable, and verifiable.
Getting Started: One Config Block on the Tap, Zero Changes Downstream
The goal of this has never been speed for its own sake. It’s about making database replication pipelines finish comfortably inside their SLA windows without asking data engineers to learn a new mental model. Enable BATCH on your tap with one consistent config block and everything downstream just works faster.
If you’re replicating from MySQL or Postgres today, upgrade to the latest tap-mysql or grab the new Arrow-enabled tap-postgres, pair it with target-snowflake and see the difference for yourself.