Open-weight models are increasingly important. Guardrails on frontier models affect cybersecurity research are pushing us toward open-weight LLMs. But are you sure the quant you downloaded is safe?
Trojan models remain largely theoretical because they are difficult to identify. Triggers may be a date, codebase, environment, or pattern; malicious behaviour may include injecting vulnerable code, overlooking vulnerabilities, or favouring a dependency. An unknown trigger and payload create a challenging problem.
Most supervised fine-tuning methods for creating Trojans miss triggers, break normal tasks, or leak payloads without a trigger. We developed a robust pipeline for realistic Trojan models with specific, effective triggers that preserve standard capabilities.
Using a zoo of Trojan models, we test methods from linear probes to J-space across triggers, payloads, model sizes, and architectures. Our results provide a foundation for detecting and mitigating Trojans in open-weight LLMs.