DialSummEval: Revisiting Summarization Evaluation for Dialogues

Mingqi Gao, Xiaojun Wan

Abstract

Dialogue summarization presents unique challenges compared to traditional document summarization, yet evaluation often relies on outdated metrics like ROUGE. This paper introduces DialSummEval, a comprehensive benchmark of human judgments across 14 models. The study reveals that ROUGE correlates poorly with human intuition in dialogue contexts. Instead, question-answering (QA) based metrics and model-based scores like BARTScore offer significantly more reliable assessments of factual consistency and relevance.

Introduction

In Natural Language Processing, summarization research has historically focused on structured text like news articles. However, dialogue summarization—covering chats, meetings, and interviews—is increasingly critical. Dialogues are inherently informal, involve multiple speakers, and require sophisticated reasoning to resolve coreferences and speaker intent.

The standard evaluation metric, ROUGE, measures n-gram overlap between a generated summary and a human reference. In the context of dialogue, ROUGE is often "blind" to critical errors:

  1. Factual Inconsistency: Misrepresenting dates, times, or actions.
  2. Coreference Errors: Misattributing actions to the wrong speaker or using incorrect pronouns.
  3. Coherence: Failing to produce a logically flowing narrative from fragmented chat logs.
Figure 1: Examples of dialogue summarization where ROUGE scores fail to capture factual errors, such as misattributing who will call whom.

The DialSummEval Benchmark

To address the lack of reliable evaluation, the authors conducted a large-scale meta-evaluation. They didn't propose a new model but rather "evaluated the evaluators."

Methodology

The researchers sampled 100 dialogues from the SAMSum dataset (a collection of 16k messenger-like conversations). They generated summaries using 14 different models, ranging from simple baselines (LEAD-3) to state-of-the-art transformers (BART, PEGASUS).

Human annotators then rated these outputs across four key dimensions:

  • Coherence: The logical flow and structure of the summary.
  • Consistency: Factual alignment with the source dialogue.
  • Fluency: Grammatical correctness and naturalness.
  • Relevance: Inclusion of the most important information.

The resulting dataset, DialSummEval, contains 12,600 individual human ratings, providing a robust "gold standard" for testing automatic metrics.

Metric Performance and Correlation

The core of the study involved testing 18 categories of automatic metrics against human judgments. These included traditional n-gram overlap (ROUGE, BLEU), embedding-based methods (BERTScore), QA-based approaches (QuestEval, FEQA), and entailment-based models (FactCC).

Figure 3: Correlation between automatic metrics and human judgments. QA-based metrics like QuestEval and model-based scores like BARTScore show higher correlation with human ratings than ROUGE.

Key Findings

  • The ROUGE Ceiling: Increasing the "n" in ROUGE-n (e.g., ROUGE-2 vs ROUGE-L) does not improve its correlation with human judgment in dialogues. It remains a poor proxy for quality.
  • QA-Based Superiority: Metrics like QuestEval emerged as the most reliable. By generating questions from the summary and answering them using the source text, these metrics prioritize factual accuracy over word-matching. QuestEval showed strong system-level correlations for Consistency ($r=0.85$) and Relevance ($r=0.83$).
  • Model-Based Metrics: BARTScore and BERTScore consistently outperformed ROUGE and BLEU across nearly all human dimensions.
  • Specialized vs. General Models: Generic pre-trained models like BART often outperformed models specifically architected for dialogue. This suggests that the underlying reasoning capabilities of large language models (LLMs) are more vital than task-specific tweaks.

Conclusion and Future Work

DialSummEval demonstrates that the dialogue summarization community must move beyond ROUGE. While QA-based and model-based metrics are more computationally expensive, their ability to mirror human intuition makes them essential for meaningful progress.

Future research should explore "reference-less" evaluation—metrics that assess a summary directly against the source dialogue. This approach bypasses the inherent flaws and biases sometimes found even in human-written "gold standard" references.

Summary updated

Abstract

Dialogue summarization is receiving increasing attention from researchers due to its extraordinary difficulty and unique application value. We observe that current dialogue summarization models have flaws that may not be well exposed by frequently used metrics such as ROUGE. In our paper, we re-evaluate 18 categories of metrics in terms of four dimensions: coherence, consistency, fluency and relevance, as well as a unified human evaluation of various models for the first time. Some noteworthy trends which are different from the conventional summarization tasks are identified. We will release DialSummEval, a multi-faceted dataset of human judgments containing the outputs of 14 models on SAMSum.
BibTeX
@inproceedings{gao2022dialsummeval,
  author = {Mingqi Gao and Xiaojun Wan},
  title = {{DialSummEval: Revisiting Summarization Evaluation for Dialogues}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.418},
}
Figure