Petr Simecek
AI Researcher @ AISLE — making LLM scanner failures measurable
Actions
Biostatistician by training, time-series crystal-ball operator at Google and protein-knot entangler at Masaryk University. Petr now teaches cheap LLMs to distrust their own vulnerability reports at AISLE. This experiment has somehow put his name on 37 CVEs across OpenSSL, libpng, PostgreSQL, OpenEMR, and OpenClaw. The bugs apparently did not check his résumé. Co-author of HoF-Bench and an ICML 2026 workshop paper on stochastic scanner failures.
Links
The Scanner Said Clean. We Ran It Again: Stochastic False Negatives in LLM Vulnerability Detection
An LLM vulnerability scanner can miss a real bug on one run and find it on the next without signaling uncertainty.
We measured this failure in 25,000 trials across 50 models on a benchmark selected for on-and-off detection. A separate 7,600-run study tested rediscovery of 95 disclosed CVEs in OpenSSL, curl, GnuTLS and five other heavily audited projects.
The results challenge the usual "pick the strongest model" strategy: repeated passes help, but diverse inexpensive models cover complementary blind spots and outperform a costlier single model. We will show our open-source scanner finding - and missing - the same bug, reveal which intuitive pipeline additions failed to justify their cost, and examine the gap between broadly detectable JS and PHP bugs and stubbornly hard stateful C vulnerabilities. Attendees will learn how to measure cumulative detection, compare complementary coverage and identify cases needing other techniques, using the public benchmark, harness and scanner.
Petr Simecek
AI Researcher @ AISLE — making LLM scanner failures measurable
Links
Actions
Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.
Jump to top