TL;DR
While LLM-based guard models are widely deployed to filter out harmful prompts and responses, this paper reveals they suffer from severe overconfidence and miscalibration, especially when facing jailbreak attacks. By evaluating 9 prominent guard models across 12 benchmarks, the authors demonstrate that post-hoc calibration techniques—specifically temperature scaling and contextual calibration—can significantly improve the reliability of content moderation without retraining the models.
What Problem Does This Paper Address?
To keep AI systems safe, developers use "guard models" (like Llama-Guard) to monitor conversations. These models perform two primary tasks: prompt classification (checking if a user's input is safe) and response classification (checking if the AI's output is safe), as illustrated in Figure 1.
While current guard models achieve high classification accuracy (F1 scores), their confidence calibration has been largely ignored. In high-risk deployment scenarios, it isn't enough for a guard model to just output "safe" or "unsafe"—we need its probability score to reflect reality. If a model blocks a prompt with "99% confidence," it should actually be right 99% of the time. If the model is poorly calibrated, it can confidently let harmful content slip through or block benign user queries due to minor domain shifts.
Methods
The authors systematically measure calibration using Expected Calibration Error (ECE), which quantifies the gap between a model's predicted confidence and its actual accuracy. Because instruction-tuned LLMs do not naturally output standard probabilities, the authors extract the logits of the target safety tokens (e.g., "safe" vs. "unsafe") and re-normalize them.
To fix miscalibration without the massive computational expense of retraining, they explore three post-hoc calibration techniques:
- Temperature Scaling (TS): A standard baseline that uses a validation set to find a scalar parameter $T$ to smooth or sharpen the probability distribution.
- Contextual Calibration (CC): A zero-shot approach originally designed to mitigate LLM prompt bias. It estimates test-time bias by feeding the model content-free inputs (like a single space token) and adjusting the prediction weights accordingly.
- Batch Calibration (BC): An approach that infers contextual bias from a batch of unlabeled test samples rather than content-free tokens.
Datasets
The study evaluates models across 12 distinct public benchmarks:
- Prompt Classification: OpenAI Moderation, ToxicChat Test, Aegis Safety Test, SimpleSafetyTests, XSTest, Harmbench Prompt, and WildGuardMix Test Prompt.
- Response Classification: BeaverTails Test, SafeRLHF Test, Harmbench Response, WildGuardMix Test Response, and the adversarial Harmbench-adv set.
Key Insights and Ideas
- The Overconfidence Epidemic: Across almost all datasets, guard models cluster their predictions heavily between 90% and 100% confidence, even when they are completely wrong.
- Jailbreaks Destroy Calibration: Under adversarial jailbreak attacks (like GCG or AutoDAN), prompt classification calibration degrades drastically. Interestingly, as shown in Figure 3, miscalibration is far worse for prompts than for responses, likely because jailbreak text introduces a massive distribution shift that guard models fail to handle gracefully.
- Misaligned Objectives: The authors discovered a fascinating vulnerability: guard models tasked with evaluating a response are heavily biased by the prompt. When given an unsafe prompt paired with a completely blank/safe response, models like Llama-Guard still confidently flag the response as "unsafe."
- Contextual Calibration is a Hidden Gem: For prompt classification, Contextual Calibration (CC) works surprisingly well. This is a major win because CC does not require a validation set, making it highly practical for "in-the-wild" deployments where validation data is unavailable.
Key Results
The baseline evaluation showed that even the best models exhibit poor calibration, with average ECE values frequently exceeding 10% (lower is better). WildGuard achieved the lowest average ECE for prompt classification (14.4%), while MD-Judge led response classification (11.4%).
Reliability diagrams visually confirm this overconfidence; the models' accuracy consistently falls below the ideal diagonal calibration line:
When applying calibration fixes:
- For Prompts: Contextual Calibration (CC) emerged as the top performer. For instance, it slashed the prompt ECE of Llama-Guard2 from 22.9% down to 20.0% on average, and yielded massive improvements on specific datasets like Harmbench.
- For Responses: Temperature Scaling (TS) was more effective than CC. Response evaluation involves long, multi-sentence contexts where simple content-free tokens fail to accurately capture the model's underlying bias.
Limitations and Future Work
The primary limitation is that these post-hoc methods require access to token-level logits, meaning they cannot be applied to closed-source, API-only moderation tools (like OpenAI's Moderation API). Furthermore, no single calibration method solved all issues across every dataset, highlighting a trade-off between methods depending on whether you are auditing prompts or responses.
The authors advocate that future guard models must include confidence calibration metrics in their standard evaluation suites, rather than relying solely on F1 or accuracy scores.
Summary updated