CREAM: Comparison-Based Reference-Free ELO-Ranked Automatic Evaluation for Meeting Summarization

Ziwei Gong, Lin Ai, Harshsaiprasad Deshpande, Alexander Johnson, Emmy Phung, Zehui Wu, Ahmad Emami, Julia HirschbergarXiv 2024 · arXiv

TL;DR

Evaluating long-context meeting summaries is notoriously difficult for LLMs due to severe self-bias and the "middle curse." To solve this, CREAM introduces a reference-free, comparison-based evaluation framework that extracts key facts from candidate summary pairs and ranks them using an Elo rating system, achieving a perfect 1.0 correlation with human preferences.

What Problem Does This Paper Address?

Summarizing long, multi-speaker meeting transcripts is a highly valuable but challenging NLP task. While Large Language Models (LLMs) are increasingly used to automate summary evaluation, existing absolute-scoring metrics (like G-Eval) break down in long-context dialogue scenarios.

The authors identify three critical flaws in current LLM evaluators:

  1. High Self-Bias: LLMs consistently overrate their own generated summaries, often assigning them near-perfect scores.
  2. The "Middle Curse": Evaluators struggle to track information accurately across lengthy transcripts, frequently missing or misrepresenting details in the middle sections of a text.
  3. Factuality False Alarms: In real-world datasets, true factual errors are rare and usually stem from Automatic Speech Recognition (ASR) glitches. However, standard LLM evaluators frequently hallucinate "out-of-context" factuality errors, leading to high false-positive rates.

Consequently, the authors argue that automatic evaluation for meeting summaries should pivot away from absolute factuality scoring and focus instead on robustly measuring completeness and conciseness via relative comparisons.

Methods

CREAM (Comparison-based Reference-free Elo-ranked Automatic evaluation for Meeting summarization) bypasses the need for gold-standard human reference summaries or massive source transcripts during the evaluation phase.

Figure 1: Illustrations of current evaluation framework (left) and the CREAM framework (right). On the left, traditional methods independently summarize and score each meeting transcript against reference texts. On the right, CREAM distills candidate summary pairs into key facts, conducts pair wise comparison of summary pairs, and then uses an Elo rating system to rank summaries based on their relative quality.

The framework operates in three main stages:

  1. Key Fact Extraction: Instead of extracting facts from a massive transcript, CREAM concatenates two candidate summaries ($S_m$ and $S_n$) into a single source paragraph. An LLM uses Chain-of-Thought (CoT) zero-shot prompting to distill this text into a concise, non-redundant list of atomic "key facts" (typically capped at 16 or 30).
  2. Key Fact Alignment: The LLM evaluates each summary against the extracted key facts, determining whether a fact is supported and mapping it to specific line numbers in the summary.
  3. Elo-Based Ranking: Pairwise win/loss/draw scores for completeness and conciseness are calculated from the alignment step. These scores are fed into a standard Elo rating system (commonly used in chess and competitive gaming) to dynamically compute a global quality ranking across all evaluated models.

Datasets

The authors benchmarked their approach using both public and private datasets:

  • QMSum: A query-based, long meeting summarization dataset containing 1,808 query-summary pairs from 232 meetings (averaging 556.8 turns) spanning product, academic, and committee discussions.
  • Internal Zoom Meeting Summarization (IZMS): A private dataset consisting of 139 real-world examples from internal corporate Zoom meetings (10 product meetings, 40 general meetings) used to evaluate factuality and key-fact extraction.
  • FRANK & REALSumm: Shorter, news-style summarization datasets (based on CNN/DM and XSum) used primarily to validate baseline LLM performance on simpler, short-context tasks.

Key Insights and Ideas

  • Pairwise Relativity Beats Absolute Scoring: Asking an LLM to assign an absolute score (e.g., 1-5) to a long summary results in compressed variance and high self-bias. Forcing the LLM to perform a direct head-to-head comparison between two summaries yields much sharper, more reliable signals.
  • Reference-Free is Viable: By extracting key facts directly from the combined candidate summaries rather than the original transcript, CREAM drastically reduces token consumption and eliminates the need for human-annotated gold references.
  • Model Strengths Differ by Metric: Through the CREAM framework, the authors discovered that OpenAI's GPT-4o excels at completeness (capturing more details), while GPT-4 is superior at conciseness (avoiding fluff). However, all tested models still struggle to perfectly balance the two metrics.

Key Results

When evaluated on the long-context QMSum dataset, traditional absolute-scoring LLM evaluators showed a weak correlation with human gold standards (Pearson’s $r \approx 0.5$).

By contrast, the CREAM framework achieved a perfect correlation (Pearson’s $r = 1.0$) with human preference rankings for both completeness and conciseness.

Evaluation of current evaluators on completeness and conciseness for meeting summarization, showing high self-bias.

As shown in the baseline experiments (Table 1), standard models evaluating their own outputs (the "Machine summary" column) consistently hit a deceptive 100% completeness score, a failure mode that CREAM successfully avoids by utilizing relative Elo tracking.

Limitations and Future Work

  • Evaluator Bias: The CREAM framework experiments relied exclusively on GPT-4o as the core evaluator model. The authors note that the framework's judgments could be subject to biases inherent to GPT-4o's training data, and future work should validate CREAM using diverse open-source and proprietary LLM evaluators.
  • Data Privacy Constraints: The real-world corporate data (IZMS) used to prove the industrial utility of the framework cannot be publicly released due to confidentiality agreements.
  • Future Directions: The authors suggest that CREAM's automated, reference-free reward signal makes it an ideal candidate for integration into Reinforcement Learning from AI Feedback (RLAIF) pipelines to iteratively optimize summarization models.

Summary updated

Abstract

Large Language Models (LLMs) have spurred interest in automatic evaluation methods for summarization, offering a faster, more cost-effective alternative to human evaluation. However, existing methods often fall short when applied to complex tasks like long-context summarizations and dialogue-based meeting summarizations. In this paper, we introduce CREAM (Comparison-Based Reference-Free Elo-Ranked Automatic Evaluation for Meeting Summarization), a novel framework that addresses the unique challenges of evaluating meeting summaries. CREAM leverages a combination of chain-of-thought reasoning and key facts alignment to assess conciseness and completeness of model-generated summaries without requiring reference. By employing an ELO ranking system, our approach provides a robust mechanism for comparing the quality of different models or prompt configurations.
BibTeX
@misc{gong2024cream,
  author = {Ziwei Gong and Lin Ai and Harshsaiprasad Deshpande and Alexander Johnson and Emmy Phung and Zehui Wu and Ahmad Emami and Julia Hirschberg},
  title = {{CREAM: Comparison-Based Reference-Free ELO-Ranked Automatic Evaluation for Meeting Summarization}},
  year = {2024},
  eprint = {2409.10883},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.10883},
}

Loading related content...

Figure