Session
Inside a Change Data Capture Connector: Memory, Batching, and the JVM
"Just read the transaction log" is the standard advice for change data capture, and it is good advice that hides a decade-long engineering problem. The log exists for crash recovery, not for consumers: changes from concurrent transactions are often interleaved, rollbacks appear after the fact, and the position you can safely resume from is not the position you last read.
This talk follows a transaction from its commit to a delivered event, drawn from years of maintaining the Oracle connector in the Debezium project, an open-source change data capture platform. Oracle is the working example because every hard problem surfaces at once, and each one has a close equivalent in every other database that uses transaction logs. Everything runs on the JVM, and most of the hard parts turn out to be JVM problems: memory, batching, and I/O through a JDBC driver.
We'll cover:
- Buffering changes until commit: heap versus off-heap and disk-backed caches
- Garbage collection pressure from long transactions, and what it costs each cache type
- Reading Oracle in chunks with JDBC: batch sizing, result set streaming, incomplete windows, and resuming without dropping or duplicating events
- What an offset really is: low-watermark versus last-read positions, and why a naive restart replays or skips
- DDL landing mid-stream: schema history, and why the connector keeps its own copy of the catalog instead of relying on the database
- Long-running active transactions and the retention window: when the database has discarded what you need and the JMX metrics that warn you first
No DBA experience required. You'll leave understanding why CDC tools behave the way they do, and which questions to ask of any of them before you trust one with your production data.
Chris Cranford
Principal Software Engineer, IBM - Debezium maintainer
Charlotte, North Carolina, United States
Links
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top