Event-driven architectures love short, snappy consumers — but what happens when a single job takes six hours, must never run twice, and is picky about its runtime version?

We faced exactly this: a monolithic MATLAB execution platform that couldn't scale, couldn't run versions in parallel, and took the whole system down whenever one calculation ran out of memory. We rebuilt it as an event-driven system on Kubernetes, using Kafka as the scheduling backbone for heavy, long-living compute jobs.

In this talk, we walk through the production architecture and the hard-won lessons behind it:

- How to trigger hours-long jobs through a single Kafka topic without blocking the consumer group — and why partition count becomes your scaling contract
- A requeue-with-TTL pattern that lets runners decline work they can't handle (wrong version, already busy, shutting down) and routes exhausted records to a dead-letter topic
- Exactly-once execution semantics that guarantee every job — whether it runs for seconds or hours — is executed once and only once
- Graceful shutdown: making long-running jobs respect Kubernetes SIGTERM instead of fighting it
- Decoupling payloads from events with object storage, and streaming job state, logs, and control commands over dedicated topics

You'll leave with a concrete, battle-tested blueprint for using Kafka to orchestrate workloads that are far heavier and longer-lived than the typical streaming use case — applicable to ML training, simulations, batch analytics, or any compute that refuses to fit in a 30-second consumer callback.

**Track fit:** Scalability & reliability / Production case studies (Kafka)