Comprehensiveness Metrics for Automatic Evaluation of Factual Recall in Text Generation

Adam Dejl, James Barry, Alessandra Pascale, Javier Carnerero-Cano

TL;DR

While the AI community has spent years fighting "hallucinations" (factual precision), we’ve largely ignored "sins of omission"—when an LLM provides a true but dangerously incomplete answer. This paper introduces three novel metrics to evaluate comprehensiveness (factual recall), finding that while complex graph-based methods offer great interpretability, a simple end-to-end LLM evaluator is surprisingly effective at catching what’s missing.

What Problem Does This Paper Address?

In safety-critical domains like medicine or law, a model that is 100% accurate but only tells half the story is just as dangerous as one that hallucinates. If you ask about a medication's side effects and the LLM only lists the mild ones while omitting a rare but fatal one, the response is "factually precise" but a failure in "comprehensiveness."

Current evaluation benchmarks (like FActScore) focus on precision—verifying if what was said is true. This paper tackles the harder "recall" problem: identifying specific pieces of information present in a reference corpus that the model failed to include in its response.

Methods

The authors propose and compare three automated evaluation strategies:

  1. NLI-Based Metric: This pipeline decomposes both the model output and the reference text into "atomic statements." It then uses Natural Language Inference (NLI) to build a directed graph where edges represent entailment. If a reference fact isn't "entailed" by any part of the model's response, it's flagged as missing.
  2. Q&A-Based Metric: Instead of comparing statements, this method "mines" questions from the reference text. It then asks the model's output and the reference text to answer those same questions. If the answers don't match or the model can't answer, a gap is identified.
  3. End-to-End (E2E) Metric: A "black box" approach where a strong LLM (like Llama 3 or GPT-4 class models) is given the prompt, the reference text, and the model's response, and is asked to directly list which facts were covered and which were missed.

Datasets

The researchers performed a "meta-evaluation" (evaluating the evaluators) using:

  • WikiContradict: A dataset of Wikipedia passages with known inconsistencies.
  • ConflictBank: A benchmark designed to see how LLMs handle conflicting information.
  • ELI5 (Explain Like I'm Five): The authors curated 500 real-world questions from Reddit, paired with web-retrieved search results, to test the metrics in a realistic Retrieval-Augmented Generation (RAG) setting.

Key Insights and Ideas

  • The "Context Gap" in NLI: The NLI-based approach, while mathematically elegant, often fails because it looks at "atomic facts" in isolation. Some facts only make sense—or only entail others—when the surrounding context is preserved.
  • The Robustness of Q&A: The Q&A-based method proved to be the most "robust" to the choice of the evaluator model. Because it forces the evaluator to answer specific questions rather than just "judge" text, it reduces the subjectivity of the evaluation.
  • Granularity vs. Efficiency: There is a clear trade-off. The NLI and Q&A methods produce a "fact graph" that shows exactly why a score was given (high interpretability), whereas the E2E method is much faster and cheaper but offers less "diagnostic" utility for developers.

Key Results

  • E2E is the Winner (Mostly): On the WikiContradict benchmark, the End-to-End approach achieved the highest Label Match Rate (LMR) of 0.85, proving that modern LLMs are remarkably good at spotting omissions when prompted correctly.
  • Model Performance: When evaluating popular open-weight models on the ELI5 dataset, gpt-oss-120b emerged as the most comprehensive, significantly outperforming smaller models in its ability to synthesize information from multiple sources.
  • Computational Intensity: The NLI method is the most expensive, requiring $O(N^2)$ comparisons between atomic facts, making it difficult to scale to very long documents.

Limitations and Future Work

The authors highlight two main hurdles:

  1. The "Source of Truth" Problem: These metrics only measure comprehensiveness relative to a provided corpus. If your reference documents are biased or incomplete, the evaluation will be too.
  2. Evaluator Bias: The metrics rely on an "evaluator LLM." If the evaluator isn't significantly smarter than the model being tested, it may miss subtle omissions or misinterpret complex logical entailments.

Future work suggests integrating these metrics directly into the training loop (e.g., via RLHF) to explicitly reward models for being thorough rather than just brief and accurate.

Summary updated

Abstract

Despite demonstrating remarkable performance across a wide range of tasks, large language models (LLMs) have also been found to frequently produce outputs that are incomplete or selectively omit key information. In sensitive domains, such omissions can result in significant harm comparable to that posed by factual inaccuracies, including hallucinations. In this study, we address the challenge of evaluating the comprehensiveness of LLM-generated texts, focusing on the detection of missing information or underrepresented viewpoints. We investigate three automated evaluation metrics: (1) an NLI-based method that decomposes texts into atomic statements and uses natural language inference (NLI) to identify missing facts, (2) a Q&A-based metric that extracts question-answer pairs and compares responses across sources, and (3) an end-to-end approach that directly identifies missing content using LLMs. Our experiments demonstrate the surprising effectiveness of the simple end-to-end metric compared to more complex metrics, though at the cost of reduced robustness, interpretability and result granularity. We further assess the comprehensiveness of responses from several popular open-weight LLMs when answering user queries based on multiple sources.
BibTeX
@inproceedings{dejl2026comprehensiveness,
  author = {Adam Dejl and James Barry and Alessandra Pascale and Javier Carnerero-Cano},
  title = {{Comprehensiveness Metrics for Automatic Evaluation of Factual Recall in Text Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1744},
}
Figure