LLMs are showing up in SOC tools, from log triage to incident summaries. But can we trust their outputs in critical workflows? This session explores the promises and pitfalls of using LLMs in security operations. We’ll evaluate real-world use cases like auto-generating detections, summarizing incidents, and helping with reverse engineering tasks. Through examples and benchmarks, we’ll explore where LLMs shine, where they hallucinate, and how to build secure, auditable pipelines around them. Attendees will leave with a framework to evaluate AI tools in the SOC, and a clear sense of when to automate, when to supervise, and when to just say no.