Abstract
Current language model evaluations reward "good test-takers" who guess blindly rather than admit ignorance, directly incentivizing the overconfidence that leads to hallucinations. This paper introduces the Distributional Correctness Score (DCS), a new metric that evaluates a model's entire probability distribution to reward honest abstention and penalize "confident incorrectness." By evaluating six prominent models across 12 benchmarks, we demonstrate that traditional accuracy masks significant underlying unreliability, with many models exhibiting systematically negative DCS scores.
Introduction: The Problem of Evaluation Pressure
Why do large language models (LLMs) hallucinate? While often attributed to training data or architecture, a significant factor is "evaluation pressure." Under traditional binary scoring (Accuracy), a correct answer receives a score of 1, while an incorrect answer or an "I don't know" (IDK) response both receive a 0. For a rational agent, the optimal strategy is to always guess, even with minimal confidence. This framework forces models to make confident assertions about information they do not possess, prioritizing "test-taking" over trustworthiness.
Existing alternatives like the Brier score aim for calibration—ensuring a model's confidence matches its success rate. However, calibration alone does not ensure trustworthiness. We require a metric that explicitly prefers a model saying "I don't know" over one that hedges its bets across several incorrect answers.
The Distributional Correctness Score (DCS)
The core contribution of this work is the Distributional Correctness Score (DCS). Unlike accuracy, which considers only the top prediction, DCS evaluates how a model distributes probability mass across all available choices.
The DCS is defined as: $$DCS = (l_c p_c - l_w P_W) \cdot (1 - p_{IDK})$$
Where:
- $p_c$: Probability assigned to the correct answer.
- $P_W$: Sum of probabilities assigned to all incorrect answers.
- $p_{IDK}$: Probability assigned to an abstention response (e.g., "I don't know").
- $l_c, l_w$: Loadings or weights for correct and incorrect answers, respectively.
This metric functions through three primary mechanisms:
- The Balance: It calculates the net belief in correctness ($p_c - P_W$). If a model is more confident in wrong answers than the right one, the score becomes negative.
- The IDK Damping: The $(1 - p_{IDK})$ term acts as a "neutral anchor." If the model puts all its mass on "I don't know" ($p_{IDK} = 1$), the score is 0. This ensures abstaining is always better than being confidently wrong, but worse than being confidently right.
- Tunable Thresholds: By adjusting $l_c$ and $l_w$, researchers can set specific "guessing thresholds," rewarding a model for guessing only when it exceeds a certain certainty level (e.g., 75%).
Experimental Evaluation
We adapted 12 established benchmarks to support DCS by adding an "I don't know" option to the evaluation sets. These benchmarks span reasoning (MMLU, ARC, GPQA), commonsense (HellaSwag, PIQA), and safety (TruthfulQA, Winogender). We evaluated six models: DialoGPT-Medium, Llama 3.2 3B Instruct, Llama TFree HAT Pretrained 7B DPO, Mistral 7B Instruct v0.3, Llama 3.1 8B Instruct, and DeepSeek R1 0528 Qwen3 8B.
Results and Analysis
Our evaluation reveals a stark contrast between traditional accuracy and distributional correctness.
As shown in the figure above, while models like Llama 3.1 8B achieve high accuracy (e.g., ~68% on MMLU), their DCS is significantly lower (0.19). This gap indicates that while the model often selects the correct answer, its underlying belief state is shaky and prone to overconfidence.
The Prevalence of Negative Scores
A startling finding is that for half of the tested benchmarks, every single model scored a negative DCS. This suggests that models are systematically more confident in their hallucinations than in the truth across many domains.
Safety-critical benchmarks like TruthfulQA and Winogender showed particularly poor performance, with models frequently distributing probability mass confidently among common misconceptions rather than abstaining. Mistral 7B Instruct v0.3 emerged as the most "trustworthy" on average (highest average DCS), though it still struggled on complex reasoning tasks.
Discussion and Limitations
DCS provides a more nuanced view of model reliability, but several areas for future work remain. Currently, DCS is designed for discrete answer sets; extending it to continuous spaces (e.g., numerical predictions) would require integrating probability density over "correct regions." Additionally, it remains an open question whether models can be steered via prompting to align their internal guessing thresholds with specific DCS loadings. Finally, choosing the appropriate values for $l_c$ and $l_w$ is a socio-technical challenge, as different applications (e.g., medical vs. trivia) require different risk profiles.
Conclusion
By shifting the focus from "getting the answer right" to "having a correct distribution of belief," DCS exposes the hidden unreliability of modern LLMs. It provides a mathematical framework to reward honesty and penalize the confident incorrectness that characterizes hallucinations.
Summary updated