Benchmarking LLMs on Vulnerability Prioritization

We present the first large-scale benchmarking of leading LLMs (GPT-4o mini, Claude 3.7, Gemini 2.5) against EPSS on the vulnerability prioritization task, using 50,000 CVEs stratified by real-world exploitation. Our results show that LLMs provide lumpy, poorly calibrated probability estimates, fail to maintain efficiency and coverage beyond 15%, and incur prohibitive inference costs at operational scale. In contrast, predictive models like EPSS and our Global Model deliver higher accuracy, better coverage, and practical cost profiles. We release our full dataset, agent (JayPT), and methodology under an MIT license to enable reproducibility and further research on scalable, evidence-driven vulnerability triage.

We present the first large-scale benchmarking of leading LLMs (GPT-4o mini, Claude 3.7, Gemini 2.5) against EPSS on the vulnerability prioritization task, using 50,000 CVEs stratified by real-world exploitation.

Cybersecurity is Ready for Local Models

Cybersecurity has standardized on global models. CVSS, EPSS, shared threat intelligence feeds. These tools represent real scientific progress and most security programs would be worse off without them. But they share a foundational assumption that doesn't hold up under scrutiny: that risk looks roughly the same across organizations.

It doesn't. An internet-facing payment processor and an air-gapped manufacturing floor share the same vulnerability databases but almost nothing else about how risk actually lands in their environments. Global scoring tells you where attacker pressure exists across the industry. It has no opinion on which of your assets matter, whether your controls are working, or which of the thousand alerts in your queue are worth acting on.

This talk is about what comes next. Local models are ML systems trained on an organization's own data: scanned assets, telemetry, remediation history, control posture. They don't replace global signals, they build on them. Global exploitation probability becomes a starting point, and local evidence updates it. The output is a risk signal tuned to your environment rather than the average of everyone else's.

We'll walk through where global heuristics consistently break down in practice, specifically in vulnerability prioritization, alert triage, and identity risk, and show how local models address each failure mode with concrete examples. We'll also get into the parts practitioners actually need to know: what data you need, how to frame the training problem, and how to tell whether your local model is actually better than what you're running today.

You'll leave with a framework for combining global and local inference, practical steps toward building and validating your own enterprise model, and a better answer to the question security teams are increasingly asking: why does this score never seem to reflect what's actually happening in my environment.

Past Exploitation Is Not a Guarantee of Future Exploitation: A Data-Driven Takedown of the KEV

There is a common assumption baked into how security teams prioritize vulnerabilities: if a CVE has ever been exploited in the wild, it belongs on the high priority list permanently. Exploited once, exploited always. The assumption feels conservative and therefore safe. The data says otherwise.

This talk presents original research into the temporal structure of vulnerability re-exploitation and what it means for how organizations should build and evaluate prioritization models. The central finding is straightforward but consequential: if a vulnerability has had any exploitation activity observed, the probability of seeing it exploited again in the next 30 days is approximately 65%. That number sounds useful. The problem is that it is not flat. It decays in a predictable and measurable way. Once three weeks pass without observed exploitation activity, that probability drops substantially. Once several months pass, it approaches baseline. Past exploitation is not a guarantee of future exploitation, and treating it as one is costing teams real prioritization accuracy.

The assumption that exploitation flips a permanent switch is wrong in a second way. Re-exploitation events are not random and not independent, but they are bursty. Most vulnerabilities see concentrated activity in short windows. Many see exploitation that stops completely and never resumes. Only a small fraction are exploited consistently and widely over time. The industry mental model, where a CVE lands on a known-exploited list and stays dangerous indefinitely, maps very poorly onto what the data actually shows. We are making prioritization decisions based on a static snapshot of a dynamic phenomenon. Prioritizing from the KEV alone means you are likely missing older vulnerabilities that attackers still find viable and have not stopped using.

Attendees will leave with a working understanding of how re-exploitation risk decays over time, a framework for incorporating recency into their own prioritization models, and a clearer sense of why the timestamp on a threat signal matters as much as the signal itself. If your current approach cannot tell the difference between a vulnerability exploited last week and one that was exploited three years ago, this talk is for you.