Why target-mssql Loads Were Stuck Row by Row
Singer targets traditionally load data one record at a time. That’s fine for small tables, but it becomes a bottleneck once you’re moving millions of rows on a tight SLA: every record means another round trip, another conversion, another chance to fall behind schedule.
target-mssql didn’t support the BATCH message capability at all. Every load, regardless of size, went through that slower record-by-record path.
The Benchmarks: 16-24x Faster Across Two Sources
Meltano’s benchmark suite tested moving data into MSSQL from two different sources, comparing the record-by-record path against the new Arrow BATCH path:
| Pipeline | Mode | Rows/sec | Speedup |
| MySQL → MSSQL | Record-by-record | 3,506.2 | N/A |
| MySQL → MSSQL | Arrow BATCH | 56,492.6 | ~16.1x faster |
| Postgres → MSSQL | Record-by-record | 2,691.8 | N/A |
| Postgres → MSSQL | Arrow BATCH | 64,096.6 | ~23.8x faster |
How Arrow BATCH Loading Works: IPC File to Temp Table, No Detours
The speedup comes from skipping per-row overhead entirely. When a BATCH message arrives with encoding: {“format”: “arrow”}, it points to one or more Arrow IPC files delivered as local file:// URIs. target-mssql reads each file directly with pyarrow.ipc.open_file and bulk-loads it into a temp table.
A few implementation details worth knowing:
- No configuration required. Any incoming Arrow-encoded BATCH message is automatically routed through the new code path.
- No blob storage detour. The BATCH path skips Azure Blob Storage staging entirely, loading straight into a temp table instead.
- Manifest files are consume-once. The target treats each file as single-use and doesn’t expect it to still exist afterward.
- Schema still comes from Singer. Table structure and typing are driven by the standard SCHEMA message exactly as before. Arrow just changes how the data gets read in, not how the table gets defined.
- Advertised in –about. target-mssql now properly advertises BATCH support so orchestrators know it’s available.
Choosing mssql-python Over adbc-driver-manager
Before writing the bulk-load path, the team evaluated two ways to get Arrow data into MSSQL: adbc-driver-manager and Microsoft’s own mssql-python driver. They went with mssql-python, which was adding native Arrow bulk-copy support, a better fit than converting Arrow data into a row iterator first.
Now Available in target-mssql v0.2.0
Arrow BATCH support ships in target-mssql v0.2.0 and later, with matching catalog and documentation updates included. If you’re already running target-mssql in a Meltano pipeline, the faster path comes with the update, no extra setup required on the target side.
Enabling it just means opting in on the tap side, with a batch_config block set to Arrow encoding. target-mssql picks up any Arrow BATCH message automatically and falls back to normal row-by-row loading for taps that don’t emit one.
Stop Letting MSSQL Loads Set Your Pipeline’s Pace
If MSSQL loads have been the slowest leg of your pipeline, this is the fix that’s been missing: no new infrastructure, no config to babysit, just a faster path that’s on by default the moment your tap emits Arrow. Update target-mssql, opt in on the tap side, and watch your load times drop.
Not on Meltano yet? This is exactly the kind of problem it’s built to solve. Give it a try on your next pipeline.
For setup details, see the Microsoft SQL Server batch ingestion guide and the Arrow BATCH Support overview for enabling Arrow on your tap.