SafeConf: A Confidence-Calibrated Safety Self-Evaluation Method for Large Language Models

Bo Zhang, Cong Gao, Linkang Yang, B Han, Minghao Hu, Zhunchen Luo, Guotong Geng, Xiaoying Bai, Junwen Zhang, Wen Yao, Zhong Wang2025 · DOI

Abstract

Bo Zhang, Cong Gao, Linkang Yang, Bingxu Han, Minghao Hu, Zhunchen Luo, Guotong Geng, Xiaoying Bai, Jun Zhang, Wen Yao, Zhong Wang. Findings of the Association for Computational Linguistics: EMNLP 2025.

2025.

Introduction

As Large Language Models (LLMs) are increasingly deployed in sensitive environments, the "LLM-as-a-judge" paradigm has become a standard for evaluating safety risks such as bias, privacy breaches, and malicious content. However, a significant reliability flaw persists: overconfidence. Models often confidently label dangerous outputs as "safe" because they lack the training to handle specific linguistic variations. SafeConf addresses this by introducing a training-based calibration approach rooted in semantic diversity, moving beyond simple probability analysis to ensure models "know what they don't know."

Methodology: Calibrating Through Semantic Mutation

The SafeConf framework improves safety self-evaluation through a three-stage pipeline:

  1. Diverse Semantic Mutation: Instead of simple repetition, a teacher model (e.g., GPT-4o mini) generates $k$ variations of a safety question across three intensity levels: Low (slight modifications), Medium (rephrasing with new perspectives), and High (complex transformations).
  2. Confidence Quantification: The target model's confidence is calculated based on its consistency across these mutated versions. A model that flips its judgment when a question is slightly rephrased receives a lower confidence score.
  3. Confidence-Thresholded Fine-tuning: Using a curated dataset of 570 items, the model is fine-tuned (via LoRA) to associate its safety judgments with these calibrated confidence levels.
Figure 1: Comparison between standard self-consistency and the SafeConf method, illustrating how semantic mutation reveals true model uncertainty.

Key Insights: The Role of Diversity

A central finding of the research is that semantic diversity—rather than mere repetition—is essential for uncovering a model's true uncertainty. The experiments demonstrated that higher-intensity mutations lead to the lowest Expected Calibration Error (ECE), aligning the model's subjective certainty with its objective correctness.

Figure 2: Impact of mutation intensity (Low, Medium, High) on Expected Calibration Error (ECE) across SafetyBench and JADE datasets.

Furthermore, the study highlights that while safety labels are helpful, incorporating the confidence score as a supervisory signal during fine-tuning provides a significantly stronger boost to self-evaluation accuracy. This efficiency is notable: state-of-the-art results were achieved with only 570 high-quality, curated samples.

Experimental Results

SafeConf was evaluated on Qwen2.5-7B-Instruct and Llama3-8B-Instruct, showing substantial improvements across several metrics:

  • Accuracy: Self-evaluation accuracy increased by an average of 5.86% for Qwen and 7.79% for Llama.
  • Calibration: ECE dropped significantly; for instance, Qwen’s ECE on SafetyBench improved from 0.2271 to 0.0509.
  • Robustness: The method reduced the Attack Success Rate (ASR) against jailbreak prompts across benchmarks like SafetyBench and JADE.
  • General Capabilities: Performance on benchmarks like MMLU and GSM8K remained stable, indicating no "catastrophic forgetting" of general knowledge.

Conclusion and Future Work

SafeConf demonstrates that targeted, diversity-driven fine-tuning can effectively calibrate LLM safety judgments without sacrificing general intelligence. While the current method requires a fine-tuning step and may face challenges with extremely complex texts, it sets a new benchmark for reliable safety self-evaluation. Future research will explore training-free variants and broader applications to diverse text types.

Summary updated

BibTeX
@misc{zhang2025safeconf,
  author = {Bo Zhang and Cong Gao and Linkang Yang and B Han and Minghao Hu and Zhunchen Luo and Guotong Geng and Xiaoying Bai and Junwen Zhang and Wen Yao and Zhong Wang},
  title = {{SafeConf: A Confidence-Calibrated Safety Self-Evaluation Method for Large Language Models}},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.186},
}
Figure