Abstract
Evaluating automatic text summarization remains a significant challenge due to the limitations of n-gram based metrics like ROUGE, which often fail to capture factual consistency and require expensive human references. We present QuestEval, a novel, reference-less metric that leverages Question Generation (QG) and Question Answering (QA) to assess both factual consistency (precision) and relevance (recall). By introducing a "question weighter" to prioritize salient information and using negative sampling for robust QA, QuestEval achieves substantially higher correlation with human judgments than traditional metrics. Notably, QuestEval outperforms ROUGE and BERTScore even when those metrics are provided with multiple human references.
Introduction
The evaluation of automatic summarization systems is critical for progress in the field. However, traditional metrics such as ROUGE rely on n-gram overlap between a generated summary and a human-written reference. These metrics suffer from several drawbacks: they correlate poorly with human qualitative assessments, they are dependent on the availability of high-quality human references, and they are largely "blind" to factual hallucinations. While recent QA-based metrics have attempted to address these issues, they typically focus on either precision or recall in isolation. QuestEval addresses these gaps by unifying both dimensions into a single, explainable framework.
Methodology
QuestEval evaluates a summary by comparing it against the source document through a series of generated questions. The framework consists of two primary components:
- Precision (Factual Consistency): Questions are generated from the summary ($S$). The metric checks if the answers derived from the summary match those derived from the source document ($D$).
- Recall (Relevance/Saliency): Questions are generated from the source document ($D$). The metric assesses whether these questions can be answered using the summary ($S$).
A key innovation is the Question Weighter ($W$), which identifies salient information in the source document to ensure that the recall component focuses on "need-to-know" facts rather than trivial details. Additionally, the QA model is trained with negative sampling to improve its ability to identify unanswerable questions, reducing the risk of hallucinated answers during evaluation. The final QuestEval score is the harmonic mean of the precision and recall scores.
Experimental Setup
The metric was validated using the SummEval benchmark, which contains 12,800 human annotations on CNN/Daily Mail summaries across four dimensions: consistency, coherence, fluency, and relevance. It was also tested on QAGS-XSUM, a dataset specifically annotated for factual correctness in highly abstractive summaries. The underlying transformer models for QA and QG were trained on SQuAD-v2 and NewsQA.
Results
QuestEval demonstrates superior performance across all evaluated dimensions. On the SummEval benchmark, it achieved an average Pearson correlation of 33.5, vastly outperforming ROUGE-1 (11.8) and BERTScore (11.8) when using a single reference.
As shown in Figure 3, QuestEval maintains a high and stable correlation with human judgments regardless of the number of references, whereas the performance of ROUGE and BERTScore degrades significantly as the number of available references decreases. Specifically, QuestEval's correlation for "Consistency" (42.0) and "Relevance" (39.2) sets a new state-of-the-art for reference-less evaluation.
Discussion and Conclusion
QuestEval offers a robust, explainable, and reference-less alternative to traditional summarization metrics. By framing evaluation as a question-answering task, it provides developers with interpretable feedback on specific factual errors or omissions. While the computational cost is higher than n-gram metrics due to the use of transformer models, the gain in human correlation and the elimination of reference dependency make it a powerful tool for summarization research. Future work will focus on cross-domain adaptation and multilingual support.
Summary updated