COVERAGE-TRACE: Source-Side Counterfactual Tracing for Detecting Omissions in LLM Summaries

Ali Karkehabadi, Jamshid Hassanpour, Rafael E. BanchsarXiv 2026 · arXiv

TL;DR

Detecting what an LLM forgot to say is much harder than catching what it made up. COVERAGE-TRACE tackles this by performing "what-if" experiments on the source text: if changing a specific fact in the source causes a summary's likelihood to drop significantly, the model likely "covered" that fact; if the likelihood stays flat, it was probably an omission.

What Problem Does This Paper Address?

In the world of LLM evaluation, we are getting quite good at spotting hallucinations—claims in a summary that aren't supported by the source. However, omissions are a different beast. An omission is a "ghost": there is no specific span of text in the output to inspect because the information simply isn't there.

Existing metrics like FactCC or AlignScore focus on support (is the output true given the input?). Completeness evaluators usually try to "recover" facts from the output using Q&A or NLI. But these methods still reason primarily from the observed output. The authors of COVERAGE-TRACE argue that to truly detect an omission, we should look at the dependence of the summary on the source. If a summary is truly "covering" a fact, its probability should be inextricably linked to that fact's presence in the source.

Methods

The researchers introduce a source-side counterfactual framework. Instead of analyzing the summary text in isolation, they perturb the source document and watch how the model reacts.

The process, illustrated in the figure below, follows four main steps:

  1. Identify a Target Fact: Pick an atomic fact in the source (e.g., a quantity like "3/4").
  2. Generate Counterfactuals: Replace that fact with type-matched alternatives (e.g., "3/4" becomes "2/3" or "1/2") while keeping the rest of the source identical.
  3. Trace Likelihood: Take a candidate summary and hold it fixed. Calculate its teacher-forced, length-normalized log-likelihood under both the original and the counterfactual sources.
  4. Aggregate and Interpret: If the summary covers the fact, its likelihood should drop significantly when the source fact changes. If the summary omits the fact, it shouldn't "care" about the change. The Multi-CF approach averages these effects across multiple counterfactuals to reduce noise.
Figure 1: COVERAGE-TRACE holds the candidate fixed and measures its likelihood under the original source and targeted counterfactual sources. Multi-CF averages intervention effects; larger omission-oriented scores indicate weaker dependence on the target fact.

Datasets

The study uses a benchmark derived from QMSum, a query-based meeting summarization dataset.

  • 80 matched pairs: Each pair consists of one "covered" summary (containing the target fact) and one "omitted" summary (identical but with the target fact removed).
  • Fact Types: The benchmark spans 24 decision rationales, 24 qualifications, 20 quantities, and 12 temporal/causal facts.
  • Models: Evaluation was performed across four open-weight models: Falcon-7B, Llama-3-8B, Qwen2.5-7B, and Qwen2.5-14B.

Key Insights and Ideas

  • Omission as Dependence: The core "aha" moment is reframing omission detection as a causal dependency problem rather than a text-matching problem.
  • The Power of Aggregation: Using a single counterfactual (Single-CF) is effective, but averaging across three (Multi-CF) consistently boosts performance and reduces sensitivity to the specific replacement chosen.
  • Target Specificity: By "shuffling" counterfactuals (applying a counterfactual for Fact A to a summary of Fact B), the authors proved the signal is highly specific. Shuffling dropped the AUROC on Falcon-7B from 0.779 to near-chance (0.533).
  • Mechanistic Localization: Using activation patching, the authors found that the "omission signal" is strongest in the very early layers of the model, suggesting that the model's internal representation of the source fact is established early in the processing pipeline.

Key Results

COVERAGE-TRACE significantly outperforms traditional text-only baselines.

  • Performance: Multi-CF achieved an AUROC of 0.764–0.825 across the four models. For comparison, the strongest text-only baseline (DeBERTa atomic NLI) reached only 0.640.
  • Likelihood vs. Text: Simply measuring the likelihood of a summary under the original source (without counterfactuals) only yields an AUROC of ~0.58–0.60, proving that the intervention is the key to the signal.
Comparison of COVERAGE-TRACE (Multi-CF) against various baseline evaluators on Falcon-7B.

Limitations and Future Work

  • Surface-Form Sensitivity: The biggest hurdle is that the metric is not yet "wording-invariant." When the authors tested meaning-preserving paraphrases of the source, the AUROC dropped to 0.59–0.62. This means the model's likelihood is still partly tied to specific lexical choices rather than pure semantics.
  • Model Access: Unlike black-box metrics, COVERAGE-TRACE requires access to the model's token-level log-probabilities, which isn't always possible with proprietary APIs (though many now offer logprobs).
  • Fact Selection: The method assumes you already know which source facts you want to check for coverage. It doesn't automatically decide which facts are "important" enough to warrant inclusion in a summary.

Future work will likely focus on making these counterfactual signals more robust to paraphrasing, potentially moving toward a truly semantic "coverage probability."

Summary updated

Abstract

Omissions are difficult to verify because missing information leaves no output span to inspect: a summary may be factually supported while still excluding a decision, qualification, quantity, or condition needed to preserve the source's meaning. We introduce COVERAGE-TRACE, a source-side counterfactual framework that evaluates omission as dependence on a designated atomic source fact. It replaces only the target fact with type-matched alternatives, holds the candidate summary fixed, and measures the resulting change in its teacher-forced, length-normalized likelihood. A summary covering the target should respond more strongly than one omitting it, while aggregation across multiple counterfactuals reduces sensitivity to replacement choice. We evaluate COVERAGE-TRACE on 80 matched covered--omitted QMSum pairs spanning several fact types and four open-weight models. Multi-CF achieves 0.764--0.825 AUROC, compared with 0.640 for the strongest evaluated text-only baseline. On Falcon-7B, targeted interventions achieve 0.779 AUROC, whereas shuffling counterfactual assignments reduces performance to 0.533; target deletion is effective while unrelated deletion is not. However, adjustment for meaning-preserving paraphrases reduces AUROC to 0.591--0.624. COVERAGE-TRACE therefore provides a practical, complementary signal of target-specific source--summary dependence, but not a wording-invariant semantic coverage probability. These results establish source-side intervention as a promising approach to omission detection while identifying semantic invariance as the central remaining challenge.
BibTeX
@misc{karkehabadi2026coverage,
  author = {Ali Karkehabadi and Jamshid Hassanpour and Rafael E. Banchs},
  title = {{COVERAGE-TRACE: Source-Side Counterfactual Tracing for Detecting Omissions in LLM Summaries}},
  year = {2026},
  eprint = {2611.25539},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25539},
}
Figure