Abstract
Dialogue summarization presents unique challenges compared to traditional document summarization, yet evaluation often relies on outdated metrics like ROUGE. This paper introduces DialSummEval, a comprehensive benchmark of human judgments across 14 models. The study reveals that ROUGE correlates poorly with human intuition in dialogue contexts. Instead, question-answering (QA) based metrics and model-based scores like BARTScore offer significantly more reliable assessments of factual consistency and relevance.
Introduction
In Natural Language Processing, summarization research has historically focused on structured text like news articles. However, dialogue summarization—covering chats, meetings, and interviews—is increasingly critical. Dialogues are inherently informal, involve multiple speakers, and require sophisticated reasoning to resolve coreferences and speaker intent.
The standard evaluation metric, ROUGE, measures n-gram overlap between a generated summary and a human reference. In the context of dialogue, ROUGE is often "blind" to critical errors:
- Factual Inconsistency: Misrepresenting dates, times, or actions.
- Coreference Errors: Misattributing actions to the wrong speaker or using incorrect pronouns.
- Coherence: Failing to produce a logically flowing narrative from fragmented chat logs.
The DialSummEval Benchmark
To address the lack of reliable evaluation, the authors conducted a large-scale meta-evaluation. They didn't propose a new model but rather "evaluated the evaluators."
Methodology
The researchers sampled 100 dialogues from the SAMSum dataset (a collection of 16k messenger-like conversations). They generated summaries using 14 different models, ranging from simple baselines (LEAD-3) to state-of-the-art transformers (BART, PEGASUS).
Human annotators then rated these outputs across four key dimensions:
- Coherence: The logical flow and structure of the summary.
- Consistency: Factual alignment with the source dialogue.
- Fluency: Grammatical correctness and naturalness.
- Relevance: Inclusion of the most important information.
The resulting dataset, DialSummEval, contains 12,600 individual human ratings, providing a robust "gold standard" for testing automatic metrics.
Metric Performance and Correlation
The core of the study involved testing 18 categories of automatic metrics against human judgments. These included traditional n-gram overlap (ROUGE, BLEU), embedding-based methods (BERTScore), QA-based approaches (QuestEval, FEQA), and entailment-based models (FactCC).
Key Findings
- The ROUGE Ceiling: Increasing the "n" in ROUGE-n (e.g., ROUGE-2 vs ROUGE-L) does not improve its correlation with human judgment in dialogues. It remains a poor proxy for quality.
- QA-Based Superiority: Metrics like QuestEval emerged as the most reliable. By generating questions from the summary and answering them using the source text, these metrics prioritize factual accuracy over word-matching. QuestEval showed strong system-level correlations for Consistency ($r=0.85$) and Relevance ($r=0.83$).
- Model-Based Metrics: BARTScore and BERTScore consistently outperformed ROUGE and BLEU across nearly all human dimensions.
- Specialized vs. General Models: Generic pre-trained models like BART often outperformed models specifically architected for dialogue. This suggests that the underlying reasoning capabilities of large language models (LLMs) are more vital than task-specific tweaks.
Conclusion and Future Work
DialSummEval demonstrates that the dialogue summarization community must move beyond ROUGE. While QA-based and model-based metrics are more computationally expensive, their ability to mirror human intuition makes them essential for meaningful progress.
Future research should explore "reference-less" evaluation—metrics that assess a summary directly against the source dialogue. This approach bypasses the inherent flaws and biases sometimes found even in human-written "gold standard" references.
Summary updated