TL;DR
Detecting what an LLM forgot to say is much harder than catching what it made up. COVERAGE-TRACE tackles this by performing "what-if" experiments on the source text: if changing a specific fact in the source causes a summary's likelihood to drop significantly, the model likely "covered" that fact; if the likelihood stays flat, it was probably an omission.
What Problem Does This Paper Address?
In the world of LLM evaluation, we are getting quite good at spotting hallucinations—claims in a summary that aren't supported by the source. However, omissions are a different beast. An omission is a "ghost": there is no specific span of text in the output to inspect because the information simply isn't there.
Existing metrics like FactCC or AlignScore focus on support (is the output true given the input?). Completeness evaluators usually try to "recover" facts from the output using Q&A or NLI. But these methods still reason primarily from the observed output. The authors of COVERAGE-TRACE argue that to truly detect an omission, we should look at the dependence of the summary on the source. If a summary is truly "covering" a fact, its probability should be inextricably linked to that fact's presence in the source.
Methods
The researchers introduce a source-side counterfactual framework. Instead of analyzing the summary text in isolation, they perturb the source document and watch how the model reacts.
The process, illustrated in the figure below, follows four main steps:
- Identify a Target Fact: Pick an atomic fact in the source (e.g., a quantity like "3/4").
- Generate Counterfactuals: Replace that fact with type-matched alternatives (e.g., "3/4" becomes "2/3" or "1/2") while keeping the rest of the source identical.
- Trace Likelihood: Take a candidate summary and hold it fixed. Calculate its teacher-forced, length-normalized log-likelihood under both the original and the counterfactual sources.
- Aggregate and Interpret: If the summary covers the fact, its likelihood should drop significantly when the source fact changes. If the summary omits the fact, it shouldn't "care" about the change. The Multi-CF approach averages these effects across multiple counterfactuals to reduce noise.
Datasets
The study uses a benchmark derived from QMSum, a query-based meeting summarization dataset.
- 80 matched pairs: Each pair consists of one "covered" summary (containing the target fact) and one "omitted" summary (identical but with the target fact removed).
- Fact Types: The benchmark spans 24 decision rationales, 24 qualifications, 20 quantities, and 12 temporal/causal facts.
- Models: Evaluation was performed across four open-weight models: Falcon-7B, Llama-3-8B, Qwen2.5-7B, and Qwen2.5-14B.
Key Insights and Ideas
- Omission as Dependence: The core "aha" moment is reframing omission detection as a causal dependency problem rather than a text-matching problem.
- The Power of Aggregation: Using a single counterfactual (Single-CF) is effective, but averaging across three (Multi-CF) consistently boosts performance and reduces sensitivity to the specific replacement chosen.
- Target Specificity: By "shuffling" counterfactuals (applying a counterfactual for Fact A to a summary of Fact B), the authors proved the signal is highly specific. Shuffling dropped the AUROC on Falcon-7B from 0.779 to near-chance (0.533).
- Mechanistic Localization: Using activation patching, the authors found that the "omission signal" is strongest in the very early layers of the model, suggesting that the model's internal representation of the source fact is established early in the processing pipeline.
Key Results
COVERAGE-TRACE significantly outperforms traditional text-only baselines.
- Performance: Multi-CF achieved an AUROC of 0.764–0.825 across the four models. For comparison, the strongest text-only baseline (DeBERTa atomic NLI) reached only 0.640.
- Likelihood vs. Text: Simply measuring the likelihood of a summary under the original source (without counterfactuals) only yields an AUROC of ~0.58–0.60, proving that the intervention is the key to the signal.
Limitations and Future Work
- Surface-Form Sensitivity: The biggest hurdle is that the metric is not yet "wording-invariant." When the authors tested meaning-preserving paraphrases of the source, the AUROC dropped to 0.59–0.62. This means the model's likelihood is still partly tied to specific lexical choices rather than pure semantics.
- Model Access: Unlike black-box metrics, COVERAGE-TRACE requires access to the model's token-level log-probabilities, which isn't always possible with proprietary APIs (though many now offer logprobs).
- Fact Selection: The method assumes you already know which source facts you want to check for coverage. It doesn't automatically decide which facts are "important" enough to warrant inclusion in a summary.
Future work will likely focus on making these counterfactual signals more robust to paraphrasing, potentially moving toward a truly semantic "coverage probability."
Summary updated