At a certain scale, your "standard" indexing strategy stops being an asset and becomes a liability. When you cross the threshold of 10 billion documents, mistakes made in shard sizing, rollover policies, and tiering strategies don't just cause latency—they cause cluster-wide outages. Managing this volume requires moving beyond basic automation to a precision-engineered Index Lifecycle Management (ILM) strategy.

This session dives into the technical "horror stories" of scaling OpenSearch clusters to the multi-billion document range. We will explore the architectural trade-offs between storage density and search performance, the nuances of hot-warm-cold-frozen tiering, and the critical role of Data Prepper and Kafka in buffering ingestion spikes. Attendees will learn how to design a resilient ILM policy that balances cost, performance, and operational sanity.