Session

From 8M to 15M Rows/Sec: Designing a Columnar Wire Protocol for Zero-Overhead Ingestion

Note: The talk is delivered by QuestDB's CTO and co-founder, Vlad, and Fast Data Advocate Javier.

For years, QuestDB ingested data via ILP (InfluxDB Line Protocol), a text-based, row-oriented format. We'd pushed it to ~8 million rows/sec. But at that scale, you're paying for overhead you shouldn't have to: UTF-8 number parsing on every value, row-by-row WAL appends, and repeated string-to-symbol-ID lookups. We designed a replacement from scratch and nearly doubled throughput to ~15 million rows/sec.
QWP (QuestDB Wire Protocol) is a binary, column-oriented ingestion protocol. Booleans are bit-packed, timestamps use Gorilla delta-of-delta compression (~1 bit per value for monotonic sequences), symbols use delta dictionaries, and the schema is hashed with XXH64 so repeated batches transmit 8 bytes instead of full column definitions.
We'll walk through the key design decisions:
Column-oriented wire format: the client groups values by column, so the server writes entire columns into the WAL via memcpy instead of row-by-row appends.
Zero-allocation WebSocket path: frame parsing operates directly on native memory buffers, masking/unmasking happens in-place, all cursors are reusable. No object allocation on the hot path.
Two-layer symbol caching: per-column and per-connection caches eliminate the repeated string lookups that dominated CPU time in the text protocol.
UDP for fire-and-forget: using Linux recvmmsg to batch-receive datagrams. Packet loss means silent data loss, and that's a documented, deliberate trade-off.
We'll show allocation profiling across ILP-over-TCP, ILP-over-HTTP, and QWP-over-WebSocket, and explain why a column-oriented wire format is a natural fit for a column-oriented storage engine. All code and the full wire specification are open source under Apache 2.0.

Javier Ramirez

Developer —and Agent— Advocate at QuestDB. Fan of open source, developer communities, and data/ML. All around happy person. He/him

Madrid, Spain

Actions

Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.

Jump to top