Introduction
Meeting summarization presents a unique set of challenges compared to traditional document summarization. Meetings are inherently non-linear, characterized by "linguistic noise" such as false starts and fillers, and involve complex multi-party dynamics where context is frequently implicit. While specialized models have been developed to handle these nuances, the field continues to rely on "off-the-shelf" evaluation metrics like ROUGE. This study investigates the "under the hood" performance of these metrics, questioning whether high scores truly reflect summary quality or merely coincidental text overlap.
Methodology
The research was conducted in three distinct phases:
- Taxonomy Development: A systematic review of literature from 2019 to 2024 identified 10 intrinsic meeting challenges (e.g., speaker dynamics, low information density) and 8 observable error types (e.g., structural disorganization, hallucination, wrong references).
- Expert Annotation: Experts annotated 175 summaries generated by five models, including domain-standard encoder-decoders (like LED) and modern large language models (LLMs like GPT-3.5), using the QMSum dataset.
- Correlation Analysis: The study calculated Point-biserial and Spearman correlations between human-identified errors and scores from 11 automatic metrics, spanning count-based (ROUGE, BLEU), model-based (BERTScore, LENS), and QA-based (QuestEval) approaches.
The QMSum Dataset
The study utilizes QMSum, a benchmark for query-based, multi-domain meeting summarization. It incorporates transcripts from three distinct sources: informal academic meetings (ICSI), staged product design sessions (AMI), and formal parliamentary committee meetings (WPCP).
Key Insights: The Metric-Human Gap
The analysis revealed several critical discrepancies between automatic metrics and human judgment:
- The "Masking" Phenomenon: Metrics often fail to penalize errors. In approximately one-third of the studied metric-error combinations, the metric either ignored the error or, paradoxically, assigned a higher score when the error was present.
- Architecture-Specific Failures: Model architecture influences error profiles. Encoder-decoder models (e.g., LED) struggle with coherence and redundancy under complex speaker dynamics. Conversely, decoder-only LLMs (e.g., Zephyr) are more prone to linguistic inaccuracies and "topic listing" without sufficient context.
- Counter-Intuitive Rewards: Some metrics are actively misleading. Perplexity (PPL) was found to favor summaries with incorrect references, likely because these errors do not disrupt local fluency. Similarly, the LENS metric showed a positive correlation with structural disorganization.
- ROUGE as a Blunt Instrument: While ROUGE-1 can penalize missing information, it remains largely unresponsive to hallucinations or reasoning errors, failing to distinguish between a "mostly right" summary and a "fluently wrong" one.
Quantitative Results
The correlation analysis highlights the unreliability of common metrics in capturing specific error types, particularly hallucinations.
As shown in the correlation data, hallucinations are often rewarded: both BLEU and QuestEval showed significant positive correlations with hallucination errors (0.35 and 0.34, respectively), meaning they often gave higher scores to summaries containing fabricated information.
Conversely, structural disorganization was the error most consistently penalized by count-based metrics, with ROUGE-L showing a strong negative correlation (-0.44 in Spearman; -0.46 in Point-biserial). This is expected, as reordering events breaks the n-gram overlaps these metrics rely on. Overall, most metrics showed weak to moderate correlations with human judgments, suggesting that no single metric currently captures the full spectrum of summary quality.
Conclusion and Future Directions
The findings underscore an urgent need for evaluation methods tailored to the complexities of dialogue. The authors suggest that the future of meeting summarization evaluation lies in composite metrics—integrating the strengths of various methods—and leveraging LLMs with expert-guided prompts to detect sophisticated errors like incorrect reasoning. By releasing their codebase and annotation guidelines, the researchers aim to foster the development of more robust evaluation pipelines for the community.
Summary updated