When we ask a human to evaluate a summary, they do not typically count matching n-grams or compute embedding cosine similarities against a reference text. Instead, they read the summary to see if the core information of the original document is preserved.
Traditional evaluation metrics like ROUGE or BERTScore rely heavily on surface-level or semantic string matching. However, these approaches often fail to capture the actual functional utility of a summary. If a summary contains completely different words but preserves the essential facts, string-matching metrics penalize it. Conversely, a summary can copy phrases directly from the source while completely distorting the factual relationships, yet still receive a high ROUGE score.
To bridge this gap, researchers have developed an alternative paradigm: Question-Answering (QA) based evaluation. The core intuition is simple: If a summary is good, a reader should be able to answer questions about the source text using only the summary. By transforming evaluation from a text-matching problem into an information-retention test, QA-based metrics measure the actual propositional content and factual density of generated text.
The Mechanics of a QA Evaluation Pipeline
A typical QA-based evaluation framework operates as a three-step automated pipeline: Question Generation, Question Answering, and Answer Verification.
[Source or Reference Text] ──> (Question Generation) ──> [Target Questions]
│
▼
[Candidate Summary] ──> (Question Answering) ──> [Predicted Answers]
│
▼
(Answer Verification) ──> Final Score
- Question Generation (QG): An automated system parses a text (either the source document or a human-written reference summary) and generates a set of fact-based questions. Early implementations, such as the APES metric proposed by Matan Eyal et al. in Question Answering as an Automatic Evaluation Metric for News Article Summarization, used cloze-style questions created by masking named entities. Modern approaches use sequence-to-sequence models to generate natural language questions targeting salient information units.
- Question Answering (QA): A pre-trained QA model is given the generated questions and tasked with finding the answers. Crucially, the model is only allowed to read the candidate summary to find these answers.
- Answer Verification: The answers extracted from the summary are compared against the "ground truth" answers from the original text. If the summary contains the necessary information, the QA model will successfully recover the correct answers, yielding a high evaluation score.
Reference-Based vs. Reference-Free Evaluation
QA-based evaluation can be configured depending on whether human-authored reference summaries are available.
Reference-Based Evaluation
In a reference-based setup, such as the QAEval metric introduced by Daniel Deutsch et al. in Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary, questions are generated directly from the human reference summary. The candidate summary is then evaluated based on how many of these reference questions it can answer. As the authors demonstrate in Understanding the Extent to which Content Quality Metrics Measure the Information Quality of Summaries, this method explicitly prioritizes the transfer of propositional content over surface-level lexical similarity, achieving high correlation with manual gold-standard evaluations like the Pyramid Method.
Reference-Free Evaluation
Human references are expensive to produce and often suffer from subjective bias. QA evaluation naturally adapts to a reference-free setting by generating questions directly from the source document.
For example, the QuestEval metric proposed by Thomas Scialom et al. in QuestEval: Summarization Asks for Fact-based Evaluation utilizes a two-way reference-less architecture to measure both recall and precision:
- Recall (Informativeness): Questions are generated from the source text and answered using the summary. This checks if the summary retains the source's key information.
- Precision (Faithfulness): Questions are generated from the summary and answered using the source text. If a question generated from the summary cannot be answered by the source, it reveals that the summary contains hallucinations or factual errors.
Why It Works: The Power of Functional Verification
The primary reason QA-based evaluation correlates so strongly with human judgment is that it treats text as a knowledge base rather than a string of characters.
Consider how the system handles paraphrasing. If a source text says "The company experienced a rapid financial downturn" and the summary states "The firm lost money quickly", an n-gram metric like ROUGE will penalize the summary for a lack of token overlap. However, if a QG model generates the question "What happened to the company's finances?", a QA model reading the summary can easily extract "lost money quickly".
Furthermore, the strictness of the verification step can be tuned. In Benchmarking Answer Verification Methods for Question Answering-Based Summarization Evaluation Metrics, Daniel Deutsch et al. discovered that while advanced neural verification models are effective at recognizing semantic equivalence, the mere ability of a QA model to extract any coherent answer from a summary serves as an incredibly strong proxy for summary quality and faithfulness.
To prevent the metric from being flooded by trivial questions (e.g., asking about minor grammatical filler), recent work has focused on question selection. In Which questions should I answer? Salience Prediction of Inquisitive Questions, Yating Wu et al. introduced methods to predict the "salience" of questions. By weighting the evaluation score based on how important a question is to a human reader's core curiosity, the metric ensures that summaries are rewarded for retaining critical plot points rather than peripheral details.
Beyond Evaluation: Training Summarizers with QA Rewards
Because QA metrics provide a granular, differentiable-ready proxy for human comprehension, they can be directly integrated into model training via Reinforcement Learning (RL).
Instead of training a summarization model to maximize word-level cross-entropy or ROUGE scores—which leads to repetitive, formulaic copying—the model can be rewarded based on the "answerability" of its outputs.
- Kristjan Arumae et al. demonstrated this in Reinforced Extractive Summarization with Question-Focused Rewards and Guiding Extractive Summarization with Question-Answering Rewards, using cloze questions derived from human abstracts to guide extractive summarizers toward high-utility sentences.
- Thomas Scialom et al. expanded this to abstractive models in Answers Unite! Unsupervised Metrics for Reinforced Summarization Models, showing that QA-driven rewards yield summaries that humans find significantly more readable and factually accurate.
- Taking this concept to its logical conclusion, Weixuan Wang et al. developed the SummQ framework in Learning to Summarize by Learning to Quiz: Adversarial Agentic Collaboration for Long Document Summarization. This framework sets up an adversarial multi-agent game: one agent generates a comprehensive quiz from the source text, and a summarizer agent must iteratively refine its summary until an examinee agent can pass the quiz using only that summary.
Broad Applications across Domains
The functional nature of QA evaluation makes it highly adaptable to complex, non-traditional compression tasks where standard text metrics completely break down:
- Consumer Reviews: In Evaluation of Review Summaries via Question-Answering, Nannan Huang et al. designed the RunQA metric to verify opinion consistency, ensuring that summaries capture the specific sentiments and nuances of user reviews rather than just matching keywords.
- Long-Context Documents: Evaluating summaries of 5,000+ word documents is notoriously difficult. As highlighted by Marta R. Costa-jussà et al. in the LCFO: Long Context and Long Form Output Dataset and Benchmarking project, closed-loop information-retention tests provide an objective measure of content coverage that scales effectively where n-gram metrics fail.
- Multimodal Compression: When compressing a 20-page scientific paper into a single visual poster, Wei Pang et al. utilized an automated QG-QA cycle in Paper2Poster: Towards Multimodal Poster Automation from Scientific Papers to quantify how much knowledge was successfully retained across modalities.
By shifting the paradigm from lexical alignment to information utility, QA-based evaluation provides a robust, scalable, and intuitive framework that evaluates text generation the same way humans do: by testing what it actually teaches the reader.
Key Foundational Work
The TIPSTER SUMMAC Text Summarization Evaluation (1999) — contributed 4 concept elements to this direction.
Relative evaluation of informativeness in machine generated summaries (2007) — contributed 2 concept elements to this direction.