On Calibration of LLM-based Guard Models for Reliable Content Moderation

Hongfu Liu, Hengguan Huang, Xiangming Gu, Hao Wang, Ye Wang

TL;DR

While LLM-based guard models are widely deployed to filter out harmful prompts and responses, this paper reveals they suffer from severe overconfidence and miscalibration, especially when facing jailbreak attacks. By evaluating 9 prominent guard models across 12 benchmarks, the authors demonstrate that post-hoc calibration techniques—specifically temperature scaling and contextual calibration—can significantly improve the reliability of content moderation without retraining the models.

What Problem Does This Paper Address?

To keep AI systems safe, developers use "guard models" (like Llama-Guard) to monitor conversations. These models perform two primary tasks: prompt classification (checking if a user's input is safe) and response classification (checking if the AI's output is safe), as illustrated in Figure 1.

Figure 1: An overview of LLM-based guard models for content moderation. Guard models monitor the input and output during conversations between the user and LLM (Agent), providing a binary prediction followed by a specific unsafe content category if unsafe content is detected. The instruc- tion examples for prompt classification and response classification from LLama-Guard are detailed in the right yellow boxes respectively.

While current guard models achieve high classification accuracy (F1 scores), their confidence calibration has been largely ignored. In high-risk deployment scenarios, it isn't enough for a guard model to just output "safe" or "unsafe"—we need its probability score to reflect reality. If a model blocks a prompt with "99% confidence," it should actually be right 99% of the time. If the model is poorly calibrated, it can confidently let harmful content slip through or block benign user queries due to minor domain shifts.

Methods

The authors systematically measure calibration using Expected Calibration Error (ECE), which quantifies the gap between a model's predicted confidence and its actual accuracy. Because instruction-tuned LLMs do not naturally output standard probabilities, the authors extract the logits of the target safety tokens (e.g., "safe" vs. "unsafe") and re-normalize them.

To fix miscalibration without the massive computational expense of retraining, they explore three post-hoc calibration techniques:

  1. Temperature Scaling (TS): A standard baseline that uses a validation set to find a scalar parameter $T$ to smooth or sharpen the probability distribution.
  2. Contextual Calibration (CC): A zero-shot approach originally designed to mitigate LLM prompt bias. It estimates test-time bias by feeding the model content-free inputs (like a single space token) and adjusting the prediction weights accordingly.
  3. Batch Calibration (BC): An approach that infers contextual bias from a batch of unlabeled test samples rather than content-free tokens.

Datasets

The study evaluates models across 12 distinct public benchmarks:

  • Prompt Classification: OpenAI Moderation, ToxicChat Test, Aegis Safety Test, SimpleSafetyTests, XSTest, Harmbench Prompt, and WildGuardMix Test Prompt.
  • Response Classification: BeaverTails Test, SafeRLHF Test, Harmbench Response, WildGuardMix Test Response, and the adversarial Harmbench-adv set.

Key Insights and Ideas

  • The Overconfidence Epidemic: Across almost all datasets, guard models cluster their predictions heavily between 90% and 100% confidence, even when they are completely wrong.
  • Jailbreaks Destroy Calibration: Under adversarial jailbreak attacks (like GCG or AutoDAN), prompt classification calibration degrades drastically. Interestingly, as shown in Figure 3, miscalibration is far worse for prompts than for responses, likely because jailbreak text introduces a massive distribution shift that guard models fail to handle gracefully.
F1 score vs ECE under jailbreak attacks.
  • Misaligned Objectives: The authors discovered a fascinating vulnerability: guard models tasked with evaluating a response are heavily biased by the prompt. When given an unsafe prompt paired with a completely blank/safe response, models like Llama-Guard still confidently flag the response as "unsafe."
  • Contextual Calibration is a Hidden Gem: For prompt classification, Contextual Calibration (CC) works surprisingly well. This is a major win because CC does not require a validation set, making it highly practical for "in-the-wild" deployments where validation data is unavailable.

Key Results

The baseline evaluation showed that even the best models exhibit poor calibration, with average ECE values frequently exceeding 10% (lower is better). WildGuard achieved the lowest average ECE for prompt classification (14.4%), while MD-Judge led response classification (11.4%).

Reliability diagrams visually confirm this overconfidence; the models' accuracy consistently falls below the ideal diagonal calibration line:

Figure 2: Confidence distributions (First row) and reliability diagrams (Second row) of Llama- Guard, Llama-Guard3, Aegis-Guard-P, and WildGuard on the WildGuardMix Test Prompt set.

When applying calibration fixes:

  • For Prompts: Contextual Calibration (CC) emerged as the top performer. For instance, it slashed the prompt ECE of Llama-Guard2 from 22.9% down to 20.0% on average, and yielded massive improvements on specific datasets like Harmbench.
  • For Responses: Temperature Scaling (TS) was more effective than CC. Response evaluation involves long, multi-sentence contexts where simple content-free tokens fail to accurately capture the model's underlying bias.

Limitations and Future Work

The primary limitation is that these post-hoc methods require access to token-level logits, meaning they cannot be applied to closed-source, API-only moderation tools (like OpenAI's Moderation API). Furthermore, no single calibration method solved all issues across every dataset, highlighting a trade-off between methods depending on whether you are auditing prompts or responses.

The authors advocate that future guard models must include confidence calibration metrics in their standard evaluation suites, rather than relying solely on F1 or accuracy scores.

Summary updated

Abstract

Large language models (LLMs) pose significant risks due to the potential for generating harmful content or users attempting to evade guardrails. Existing studies have developed LLM-based guard models designed to moderate the input and output of threat LLMs, ensuring adherence to safety policies by blocking content that violates these protocols upon deployment. However, limited attention has been given to the reliability and calibration of such guard models. In this work, we empirically conduct comprehensive investigations of confidence calibration for 9 existing LLM-based guard models on 12 benchmarks in both user input and model output classification. Our findings reveal that current LLM-based guard models tend to 1) produce overconfident predictions, 2) exhibit significant miscalibration when subjected to jailbreak attacks, and 3) demonstrate limited robustness to the outputs generated by different types of response models. Additionally, we assess the effectiveness of post-hoc calibration methods to mitigate miscalibration. We demonstrate the efficacy of temperature scaling and, for the first time, highlight the benefits of contextual calibration for confidence calibration of guard models, particularly in the absence of validation sets. Our analysis and experiments underscore the limitations of current LLM-based guard models and provide valuable insights for the future development of well-calibrated guard models toward more reliable content moderation. We also advocate for incorporating reliability evaluation of confidence calibration when releasing future LLM-based guard models.
BibTeX
@inproceedings{liu2025calibration,
  author = {Hongfu Liu and Hengguan Huang and Xiangming Gu and Hao Wang and Ye Wang},
  title = {{On Calibration of LLM-based Guard Models for Reliable Content Moderation}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=wUbum0nd9N},
}
Figure