Session

Inside a Change Data Capture Connector: Memory, Batching, and the JVM

"Just read the transaction log" is the standard advice for change data capture, and it hides a multi-year engineering problem. The log exists for crash recovery, not for consumers: concurrent transactions are interleaved, rollbacks appear after the fact, and the position you can safely resume from is not the position you last read.

This talk follows one transaction from COMMIT to an event stream, drawn from years of maintaining Debezium's Oracle connector, with the PostgreSQL and MySQL equivalents throughout. Most of the hard parts turn out to be JVM problems. We'll cover:

- Buffering until commit: heap, off-heap, and disk-backed caches, and GC pressure from long transactions.
- Mining the log in windows over JDBC without dropping or duplicating events.
- What an offset really is, and why a naive restart replays or skips.
- Schema changes that land mid-stream.
- Long-open transactions, log retention, and the JMX metrics that warn you first.

No DBA knowledge required. You'll leave knowing why CDC tools behave the way they do, and what to ask before trusting one with production data.


Target audience: developers and architects who use change data capture or are evaluating it and want to understand what the tool is actually doing between a commit and an event. Intermediate. No Oracle or DBA knowledge required; every section is explained from first principles and mapped to PostgreSQL and MySQL.

Format and duration: regular 55-minute session, about 45 minutes of talk with 10 minutes for questions.

Technical requirements: slides, plus a short demo showing a transaction moving through the connector. Nothing depends on conference Wi-Fi. Projector with HDMI.

First public delivery: yes. This is a new talk and has not been given elsewhere.

Related prior talks: Change Data Streaming Patterns in Distributed Systems, Devnexus 2023. No overlap: that talk covered patterns for consumers of change events; this one covers how the events are produced.

Additional information: I maintain the Oracle connector for Debezium, an open-source project under the Commonhaus Foundation, so the failure modes described are first-hand. Oracle is the worked example because it is where every hard problem appears at once, not as a product focus; the talk is vendor-neutral. It is placed in Java & the JVM Ecosystem because the engineering problems are JVM problems: memory management for large in-flight state, off-heap and disk-backed caches, batching and streaming over JDBC, and JMX observability. It also speaks to the committee's question of which established engineering principles matter most now: bounded memory, safe restarts, and honest offsets keep any downstream consumer, including an AI pipeline, from silently missing data. It would also fit System Design.

Chris Cranford

Principal Software Engineer, IBM - Debezium maintainer

Charlotte, North Carolina, United States

Actions

Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.

Jump to top