Comprehensiveness Metrics for Automatic Evaluation of Factual Recall in Text Generation

Adam Dejl, James Barry, Alessandra Pascale, Javier Carnerero-Cano

Abstract

Large Language Model (LLM) evaluation has historically prioritized factual precision—ensuring that generated content is true. However, the inverse problem of "comprehensiveness" (factual recall) is equally critical, particularly in high-stakes environments. This paper investigates "sins of omission," where a model provides accurate but incomplete information. We propose and evaluate three metrics: NLI-based, Q&A-based, and End-to-End (E2E) evaluation. Our results demonstrate that modern LLMs, when used as end-to-end evaluators, can effectively identify missing information with high accuracy, though structured methods provide better diagnostic transparency.

Introduction

The rapid deployment of LLMs in specialized fields like medicine, finance, and law necessitates a shift in evaluation paradigms. While "hallucinations" (false claims) are well-documented, "sins of omission"—where a model fails to provide a complete picture—pose a significant risk. A model that lists only common side effects of a medication while omitting a rare, life-threatening contraindication is technically accurate but dangerously incomplete.

Example of an LLM response omitting key details in different contexts.

Current evaluation frameworks, such as FActScore, are designed to verify the precision of generated statements. This research addresses the harder problem of recall: identifying specific pieces of information present in a reference corpus that the model failed to include in its response.

Methodology

We introduce three distinct automated strategies for evaluating comprehensiveness:

  1. NLI-Based Metric: This pipeline breaks down both the model output and the reference text into "atomic statements." It employs Natural Language Inference (NLI) to construct a directed graph where edges represent logical entailment. A reference fact is considered "covered" if it is entailed by at least one part of the model's response.
  2. Q&A-Based Metric: This method "mines" potential questions from the reference text. It then prompts both the reference text and the model's output to answer these questions. A gap is identified if the model's answer contradicts the reference or if the model is unable to provide an answer.
  3. End-to-End (E2E) Metric: A high-capacity LLM (e.g., Llama 3 or GPT-4) acts as a direct judge. Given the prompt, reference text, and model response, the evaluator is tasked with listing covered and missing facts in a single pass.
Comparison of the NLI-based, Q&A-based, and End-to-End evaluation pipelines.

Experimental Setup and Results

To perform a "meta-evaluation" of these metrics, we utilized three primary datasets: WikiContradict, ConflictBank, and a curated ELI5 (Explain Like I'm Five) dataset. The latter simulates a realistic Retrieval-Augmented Generation (RAG) environment using 500 real-world Reddit questions paired with web-retrieved search results.

Our analysis revealed several key insights:

  • E2E Superiority: On the WikiContradict benchmark, the E2E approach achieved the highest Label Match Rate (LMR) of 0.85. This suggests that modern LLMs are highly capable of identifying omissions when provided with sufficient context.
  • The Robustness of Q&A: The Q&A-based method proved most resilient to the choice of evaluator model. By forcing the evaluator to perform a specific task (answering questions) rather than making a holistic judgment, it minimizes subjective bias.
  • The "Context Gap" in NLI: While theoretically sound, the NLI-based approach often struggled because atomic statements lose the surrounding context necessary for logical entailment. Furthermore, its $O(N^2)$ computational complexity makes it difficult to scale to long documents.
  • Model Benchmarking: In the ELI5 RAG setting, gpt-oss-120b emerged as the most comprehensive model, significantly outperforming smaller open-weight models in synthesizing information from multiple sources.

Limitations and Future Directions

Two primary challenges remain. First, the "Source of Truth" problem: these metrics are only as good as the reference corpus provided. Second, "Evaluator Bias": the metrics rely on an evaluator LLM that must be sufficiently sophisticated to understand complex logical relationships. Future research will focus on integrating these comprehensiveness metrics into the training loop (e.g., via RLHF) to explicitly incentivize models to be thorough and exhaustive in their responses.

Summary updated

BibTeX
@inproceedings{dejl2026comprehensiveness,
  author = {Adam Dejl and James Barry and Alessandra Pascale and Javier Carnerero-Cano},
  title = {{Comprehensiveness Metrics for Automatic Evaluation of Factual Recall in Text Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1744},
}
Figure