Chris Cranford
Principal Software Engineer, IBM - Debezium maintainer
Charlotte, North Carolina, United States
Actions
Chris Cranford is a Principal Software Engineer at IBM, previously at Red Hat, and one of the longest-serving maintainers of Debezium, the open-source change data capture project under the Commonhaus Foundation, where he serves on the extended governance committee. He reviews code across the connector portfolio, helps set the project's technical direction, is the primary maintainer of the Oracle connector, and started new efforts such as Debezium's JDBC and Elasticsearch sink connectors. Chris has contributed to Hibernate ORM, Quarkus, and Spring, has nearly three decades of experience in data management and integration across distributed systems, is a former Oracle DBA, and speaks at conferences, including Devnexus.
Links
Area of Expertise
Topics
Inside a Change Data Capture Connector: Memory, Batching, and the JVM
"Just read the transaction log" is the standard advice for change data capture, and it is good advice that hides a decade-long engineering problem. The log exists for crash recovery, not for consumers: changes from concurrent transactions are often interleaved, rollbacks appear after the fact, and the position you can safely resume from is not the position you last read.
This talk follows a transaction from its commit to a delivered event, drawn from years of maintaining the Oracle connector in the Debezium project, an open-source change data capture platform. Oracle is the working example because every hard problem surfaces at once, and each one has a close equivalent in every other database that uses transaction logs. Everything runs on the JVM, and most of the hard parts turn out to be JVM problems: memory, batching, and I/O through a JDBC driver.
We'll cover:
- Buffering changes until commit: heap versus off-heap and disk-backed caches
- Garbage collection pressure from long transactions, and what it costs each cache type
- Reading Oracle in chunks with JDBC: batch sizing, result set streaming, incomplete windows, and resuming without dropping or duplicating events
- What an offset really is: low-watermark versus last-read positions, and why a naive restart replays or skips
- DDL landing mid-stream: schema history, and why the connector keeps its own copy of the catalog instead of relying on the database
- Long-running active transactions and the retention window: when the database has discarded what you need and the JMX metrics that warn you first
No DBA experience required. You'll leave understanding why CDC tools behave the way they do, and which questions to ask of any of them before you trust one with your production data.
Your Vector Store Is Already Out of Date
When a language model answers questions from your own data using a pattern known as Retrieval-Augmented Generation (RAG), it is not reading your database. It is reading a vector index of embeddings built from the database, and that index is only as current as its last build. The data behind it lives in a relational or NoSQL datastore like PostgreSQL, Oracle, or MongoDB and changes all day, while the index refreshes at set intervals. This can lead your model to answer confidently based on stale prices, inventory, account status, or other mission-critical information. In addition, batch re-embedding scales badly: it recomputes everything to catch the small percentage of actual changes.
Change data capture (CDC) closes this gap. Debezium, an open-source CDC platform, reads database transaction logs and emits an event for every change in near real time. Recent releases add the pieces to build reliable retrieval pipelines: vector column support in PostgreSQL, MySQL, and Oracle 26ai, along with an embeddings transformation that uses a local or hosted model. Paired with Debezium Server, those changes and embeddings can be written directly to Milvus or Qdrant, with no Kafka infrastructure in between.
We'll walk through a working pipeline, all in Java: an application writing an ordinary row to PostgreSQL, Debezium Server computing embeddings in-flight and storing them in Milvus, and a Quarkus service using LangChain4j answering questions about data that changed only seconds earlier. Along the way, you'll learn about:
- Deletes and updates in a vector index, and why tombstones are vital
- Chunking and field selection for rows that don't look like documents
- Where embedding should run: in the pipeline, in the sink, or in the application
- Rebuilding the index from a fresh snapshot without stopping the stream
- Running it: where offsets live, what a restart replays, and what happens when the vector store is unavailable
No CDC background is needed. You'll leave with a design for keeping RAG in step with the system of record, and the operational checklist to run it.
Five Ways to Feed AI from Your Database, and What Breaks with Each
Every AI feature that answers from your own data depends on a path from the database to wherever the model reads: a vector index, feature store, prompt context, or an agent's tool. That path, not the model, decides whether a deleted customer is still retrievable, whether the model sees half of a transaction, and whether anyone can trace an answer back to a physical row. There are five common ways to build it, and each has a failure mode that often only shows once you're in production:
- Polling a timestamp column: misses deletes, so removed customers remain retrievable, and misses rows whose transaction committed after the poll but with an earlier timestamp
- Triggers: correct, until they put latency on every write and the DBA disables them
- Dual writes from the application: the database commits, but the publish fails, or the reverse, and now the index diverges
- The outbox pattern: transactional and clean, but something still has to relay the outbox, which lands you back at polling or the log
- Reading the actual transaction log: the only one of the five that sees deletes and every commit in order, yet the most demanding. It ties you to the database log's retention policy, so a paused consumer can lose its place; the initial load means embedding every row; schema changes arrive in the stream whether you're ready for them; delivery is at-least-once, all of which means downstream needs to tolerate replay.
This session treats the five as design choices and compares them on the guarantees that matter once a model is the consumer: deletes and the right to be forgotten; ordering and partial transactions; provenance; schema evolution; load on the source; and recovery after an outage. Freshness is on the list, but it is the easiest to notice and the least dangerous when it fails. The examples shared come from real PostgreSQL and Oracle deployments: a replication slot that filled a disk over a holiday weekend; a long-running transaction that stalled a pipeline for hours; and a column rename nobody told the pipeline about.
The model cannot tell you which of these went wrong; however, the data path can, if it was designed to do so. You'll leave with a way to choose that path for your own AI features, and a checklist for whichever approach you pick.
Chris Cranford
Principal Software Engineer, IBM - Debezium maintainer
Charlotte, North Carolina, United States
Links
Actions
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top