A Context-Aware Dual-Metric Framework for Confidence Estimation in Large Language Models

Mingruo Yuan, Shuyi Zhang, Ben KaoarXiv 2025 · DOI · arXiv

Abstract

Large Language Models (LLMs) often suffer from hallucinations and overconfidence, particularly in Contextual Question Answering (CQA) tasks where they may ignore provided evidence in favor of internal biases. We present CRUX (Context-aware entropy Reduction and Unified consistency eXamination), a framework designed to improve confidence estimation by disentangling "data uncertainty" from "model uncertainty." By contrastively sampling model outputs with and without context, CRUX measures contextual faithfulness through entropy reduction and internal stability via global consistency. Our evaluation across five diverse datasets shows that CRUX significantly outperforms existing consistency-based baselines in detecting hallucinations.

Introduction

In Contextual Question Answering (CQA), LLMs are expected to generate answers grounded in a specific document. However, these models frequently hallucinate—generating plausible but factually incorrect text—often because they rely on internal knowledge that contradicts the provided context. This "groundedness gap" makes traditional confidence estimation methods, such as self-evaluation or simple consistency checks, unreliable. A model may be perfectly consistent in providing a wrong answer if its internal bias is strong enough.

The challenge lies in determining if a model's response is truly faithful to the source material. CRUX addresses this by examining how the presence of context constrains the model's hypothesis space.

The CRUX Framework

CRUX operates through a three-stage process to evaluate the relationship between the model, the query, and the provided context.

  1. Contextual Information Gain ($\Delta H$): The framework generates two sets of answers—one using the provided context and one without it. These answers are grouped into semantic clusters using bidirectional entailment. By calculating the "entropy reduction" between these two sets, CRUX determines if the context successfully constrained the model's output. High entropy reduction indicates high contextual faithfulness.
  2. Unified Consistency (GC): To account for scenarios where context might not change entropy (e.g., the model already knew the answer or is too confused to utilize the context), CRUX measures "global consistency." This metric evaluates the stability of the model's answers across both contextual and context-free scenarios, capturing underlying model uncertainty.
  3. Neural Weighting: These two metrics are integrated using a simple 2-layer Multi-Layer Perceptron (MLP) to produce a final confidence score between 0 and 1.
Overview of the CRUX framework comparing contextual and context-free generations to estimate faithfulness and consistency.

Experimental Evaluation

We evaluated CRUX using Llama-3-8B and Qwen-14B across five benchmarks, including general CQA (CoQA, SQuAD, QuAC) and domain-specific tasks (BioASQ for biomedical and EduQG for education).

Key Results

CRUX consistently achieved the highest Area Under the Receiver Operating Characteristic curve (AUROC) compared to established baselines such as Semantic Entropy and Laplacian Eigenvalues. For instance, on the QuAC dataset with Llama-3-8B, CRUX reached an AUROC of 0.8918, outperforming lexical baselines like ROUGE-L and other semantic methods.

ROC curves for Llama-3-8B on the QuAC dataset, demonstrating CRUX's superior performance in hallucination detection.

In specialized domains, the performance was even more pronounced, with AUROC scores reaching 0.9364 on BioASQ and 0.9565 on EduQG. Ablation studies confirmed that both the contextual information gain and global consistency components are vital; removing the consistency check led to a significant drop in predictive accuracy.

Discussion and Conclusion

CRUX provides a robust method for detecting when an LLM is ignoring context, a critical step toward safer deployment in RAG (Retrieval-Augmented Generation) systems. While the framework is more computationally expensive than single-inference methods—requiring multiple samples for both contextual and context-free scenarios—the gains in reliability are substantial.

Future work includes optimizing the computational cost and extending the framework to evaluate long-form generations at a claim-by-claim level. By explicitly measuring how context reduces uncertainty, CRUX moves LLM evaluation toward a more principled, faithfulness-oriented approach.

Summary updated

BibTeX
@misc{yuan2025context,
  author = {Mingruo Yuan and Shuyi Zhang and Ben Kao},
  title = {{A Context-Aware Dual-Metric Framework for Confidence Estimation in Large Language Models}},
  year = {2025},
  eprint = {2508.00600},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.00600},
}
Figure