Abstract
Large language models (LLMs) pose significant risks due to the potential for generating harmful content or users attempting to evade guardrails. Existing studies have developed LLM-based guard models designed to moderate the input and output of threat LLMs, ensuring adherence to safety policies by blocking content that violates these protocols upon deployment. However, limited attention has been given to the reliability and calibration of such guard models. In this work, we empirically conduct comprehensive investigations of confidence calibration for 9 existing LLM-based guard models on 12 benchmarks in both user input and model output classification. Our findings reveal that current LLM-based guard models tend to 1) produce overconfident predictions, 2) exhibit significant miscalibration when subjected to jailbreak attacks, and 3) demonstrate limited robustness to the outputs generated by different types of response models. Additionally, we assess the effectiveness of post-hoc calibration methods to mitigate miscalibration. We demonstrate the efficacy of temperature scaling and, for the first time, highlight the benefits of contextual calibration for confidence calibration of guard models, particularly in the absence of validation sets. Our analysis and experiments underscore the limitations of current LLM-based guard models and provide valuable insights for the future development of well-calibrated guard models toward more reliable content moderation. We also advocate for incorporating reliability evaluation of confidence calibration when releasing future LLM-based guard models.
Introduction
As Large Language Models (LLMs) are increasingly integrated into real-world applications, ensuring their safety is paramount. "Guard models"—specialized LLMs like Llama-Guard or WildGuard—have emerged as a standard defense mechanism. These models act as safety monitors, classifying user prompts and model responses to block content that violates safety policies.
While research has traditionally focused on the accuracy of these guards, their reliability—specifically how well their confidence scores reflect the actual probability of being correct—remains under-explored. A guard model that is 99% confident but incorrect creates a dangerous false sense of security, especially when facing "out-of-distribution" data like adversarial jailbreak attacks.
Methods and Evaluation Framework
We conducted an extensive empirical study of 9 popular LLM-based guard models across 12 benchmarks. Our primary metric is Expected Calibration Error (ECE), which measures the discrepancy between a model’s predicted confidence and its observed accuracy.
To address miscalibration, we evaluated three post-hoc techniques:
- Temperature Scaling (TS): A standard method that uses a small validation set to adjust output probabilities.
- Contextual Calibration (CC): A data-free method that estimates model bias by processing "content-free" tokens (e.g., "N/A").
- Batch Calibration (BC): Estimates bias from unlabeled test batches.
The datasets spanned prompt classification (e.g., ToxicChat, XSTest), response classification (e.g., BeaverTails), and adversarial evaluation using Harmbench-adv to test resilience against jailbreak attacks like GCG and AutoDAN.
Empirical Findings: The Overconfidence Trap
Our analysis reveals that most current guard models are severely miscalibrated. They frequently exhibit an "overconfidence trap," where they output extremely high confidence scores (90-100%) even for incorrect predictions.
Key insights from our investigation include:
- Jailbreak Sensitivity: Even when models maintain high F1 scores under attack, their calibration often degrades significantly. For instance, WildGuard achieves 92.8% F1 on adversarial prompts but suffers a high ECE of 34.9%.
- Agent Diversity: Guard models are often sensitive to the specific style of the LLM they monitor. A model calibrated for Llama responses may become highly unreliable when moderating responses from Baichuan or Qwen models.
- Keyword Vulnerability: Many models rely on surface-level keywords. Simply including the word "unsafe" in a benign prompt can trigger high-confidence "unsafe" classifications, indicating a lack of deep semantic understanding.
Mitigation Strategies
We found that post-hoc calibration can significantly improve reliability:
- Contextual Calibration (CC) as a "Free Lunch": For prompt classification, CC often outperformed Temperature Scaling. Because it requires no labeled validation data, it is a highly practical solution for deployed systems.
- Temperature Scaling (TS) for Responses: For the more complex task of moderating long-form agent responses, TS remains the most effective method for reducing ECE, provided a representative validation set is available.
Conclusion
Current LLM-based guard models, while accurate, are often unreliable due to overconfidence and sensitivity to distribution shifts. We recommend that future research and model releases move beyond reporting simple accuracy metrics and include comprehensive reliability evaluations like ECE. Incorporating techniques like Contextual Calibration can provide immediate improvements in the trustworthiness of content moderation systems.
Summary updated