Session
Five Ways to Feed AI from Your Database, and What Breaks with Each
Every AI feature that answers from your own data depends on a path from the database to wherever the model reads: a vector index, feature store, prompt context, or an agent's tool. That path, not the model, decides whether a deleted customer is still retrievable, whether the model sees half of a transaction, and whether anyone can trace an answer back to a physical row. There are five common ways to build it, and each has a failure mode that often only shows once you're in production:
- Polling a timestamp column: misses deletes, so removed customers remain retrievable, and misses rows whose transaction committed after the poll but with an earlier timestamp
- Triggers: correct, until they put latency on every write and the DBA disables them
- Dual writes from the application: the database commits, but the publish fails, or the reverse, and now the index diverges
- The outbox pattern: transactional and clean, but something still has to relay the outbox, which lands you back at polling or the log
- Reading the actual transaction log: the only one of the five that sees deletes and every commit in order, yet the most demanding. It ties you to the database log's retention policy, so a paused consumer can lose its place; the initial load means embedding every row; schema changes arrive in the stream whether you're ready for them; delivery is at-least-once, all of which means downstream needs to tolerate replay.
This session treats the five as design choices and compares them on the guarantees that matter once a model is the consumer: deletes and the right to be forgotten; ordering and partial transactions; provenance; schema evolution; load on the source; and recovery after an outage. Freshness is on the list, but it is the easiest to notice and the least dangerous when it fails. The examples shared come from real PostgreSQL and Oracle deployments: a replication slot that filled a disk over a holiday weekend; a long-running transaction that stalled a pipeline for hours; and a column rename nobody told the pipeline about.
The model cannot tell you which of these went wrong; however, the data path can, if it was designed to do so. You'll leave with a way to choose that path for your own AI features, and a checklist for whichever approach you pick.
Chris Cranford
Principal Software Engineer, IBM - Debezium maintainer
Charlotte, North Carolina, United States
Links
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top