Session

The Assurance Gap: A Practical Method for Scoring AI Model Trust

A year ago, most enterprise AI ran through a frontier lab's API. That has changed fast. Open-weight models now carry the majority of AI traffic, not because they're better, but because the price gap with frontier models has become impossible to ignore, and because weights an organization already holds can't be deprecated, repriced, or quietly updated on someone else's schedule. By July 2026, Chinese-developed models held all five top spots on OpenRouter by token volume and carried more than 60% of the platform's traffic, a reversal from roughly 70% US dominance a year earlier. The shift arrived faster than most review processes could keep up, and in many organizations it arrived without security being asked first.

The problem is what's missing when these models show up: no contract, no support channel, no company whose reputation is riding on how the model behaves. An academic analysis of the largest public model repository found that roughly 60% of its 1.5 million models carry no documentation at all, and fewer than 15% publish accuracy details. When a review team goes looking for evidence, most of the time it was never produced. Internally, we measured the same gap: UltraViolet Cyber's AISec Study found 8 of 10 organizations have an approved model trust policy, while zero of ten have a fully established model registry with provenance, despite all ten attempting it. The policy exists. The engineering to enforce it doesn't.

This session introduces a method for closing that gap: a six-domain assurance score (provenance and supply chain, data handling, technical security posture, access and integration risk, operational maturity, evaluation and known behavior), each domain rated 1 to 5 against a written rubric, with an evidence-confidence label attached (unverified, documented, or independently verified) so a 4-out-of-5 built on unverified claims doesn't read the same as a 4-out-of-5 built on reproduced testing. Four gating conditions override the composite entirely, regardless of how the other domains scored. The talk closes on a worked example that produced a result we didn't expect going in: for the same model, with identical weights, the assurance score crossed a full confidence band depending on which distribution was evaluated, which is the reason this method attaches to specific artifacts rather than to model families.


No special technical requirements beyond a standard projector/screen and microphone; no live demo or internet dependency. Target audience: security engineers, AI/ML engineers, and platform teams evaluating or deploying open-weight and third-party models, as well as CISOs, security directors, and GRC leads assessing AI vendor and model risk.

John Waller

Author and Risk Advisory Practice Lead at UltraViolet Cyber

Mystic, Connecticut, United States

Actions

Please note that Sessionize is not responsible for the accuracy or validity of the data provided by speakers. If you suspect this profile to be fake or spam, please let us know.

Jump to top