MDSEval: A Meta-Evaluation Benchmark for Multimodal Dialogue Summarization

Yinhong Liu, Jianfeng He, Hang Su, Ruixue Lian, Yi Nian, Jake W. Vincent, Srikanth Vishnubhotla, Robinson Piramuthu, Saab Mansour

Introduction

Multimodal Dialogue Summarization (MDS) involves condensing conversations that include both text and images, such as those found on messaging platforms like Slack or WhatsApp. As Multimodal Large Language Models (MLLMs) advance, evaluating the quality of their summaries becomes a bottleneck. While human evaluation is the gold standard, it is not scalable. This has led to the rise of "LLM-as-a-judge" metrics. However, these automated evaluators themselves require validation. Researchers from AWS AI Labs and the University of Cambridge introduced MDSEval, a benchmark designed to meta-evaluate these automated metrics against human intuition.

The MEKI Criterion

A significant challenge in MDS evaluation is "information overlap," where the text alone provides enough context to summarize the image, allowing models to bypass true multimodal reasoning. To address this, the authors introduced the Mutually Exclusive Key Information (MEKI) criterion.

The MEKI criterion uses CLIP embeddings to ensure that images and text provide non-redundant information, forcing models to utilize both modalities.

By leveraging CLIP embeddings, MEKI filters for dialogues where the image contains information absent from the text and vice versa. This ensures that any successful summary—and any accurate evaluation of that summary—requires genuine multimodal comprehension. The paper demonstrates that MEKI scores correlate highly (ρ ≈ 0.80) with human judgments of informativeness.

The MDSEval Pipeline

The construction of MDSEval followed a rigorous four-step pipeline to ensure a high-quality, challenging benchmark for meta-evaluation.

The MDSEval pipeline: from data filtering using MEKI to multi-modal summarization, human expert evaluation, and final benchmarking of automated metrics.
  1. Data Filtering: Starting with ~57,000 dialogues from PhotoChat and DialogCC, the team applied the MEKI framework alongside consistency and length checks to produce a subset of 198 high-quality dialogues.
  2. Summarization: Four state-of-the-art MLLMs (Molmo-72B, GPT-4o-mini, Gemini 1.5 Flash, and Qwen-VL-Max) generated 990 candidate summaries using diverse prompting strategies.
  3. Human Evaluation: Three human experts graded each summary across eight dimensions, including novel metrics like Multimodal Information Balancing and Topic Progression.
  4. Benchmarking: Automated evaluators were then tested on their ability to correlate with these expert human scores.

Key Insights and Results

The benchmark revealed several critical weaknesses in current automated evaluation methods:

  • The "Hedging" Bias: Automated judges tend to concentrate their scores (often giving a "4" out of 5), failing to distinguish between "good" and "excellent" summaries. This lack of variance makes rank-based correlations less reliable than Mean Squared Error (MSE).
  • Weak Human Alignment: Spearman correlations between MLLM judges and human experts were consistently low, sometimes even approaching zero or becoming negative in specific categories.
  • Faithfulness Detection: Models struggled significantly to identify "hallucinations" or factual inaccuracies, with balanced accuracy for faithfulness detection falling below 30%.
  • Model and Positional Bias: Evaluators exhibited biases based on the order of summaries presented or favored specific model outputs regardless of quality.

Conclusion

MDSEval highlights a significant gap in our current AI infrastructure: while we can generate sophisticated multimodal summaries, our ability to automatically grade them remains limited. By providing a "ruler" grounded in the MEKI criterion, MDSEval offers a path forward for developing more robust, human-aligned evaluation metrics for the next generation of multimodal models.

Summary updated

Abstract

Multimodal Dialogue Summarization (MDS) is a critical task with wide-ranging applications. To support the development of effective MDS models, robust automatic evaluation methods are essential for reducing both cost and human effort. However, such methods require a strong meta-evaluation benchmark grounded in human annotations. In this work, we introduce MDSEval, the first meta-evaluation benchmark for MDS, consisting image-sharing dialogues, corresponding summaries, and human judgments across eight well-defined quality aspects. To ensure data quality and richfulness, we propose a novel filtering framework leveraging Mutually Exclusive Key Information (MEKI) across modalities. Our work is the first to identify and formalize key evaluation dimensions specific to MDS. We benchmark state-of-the-art modal evaluation methods, revealing their limitations in distinguishing summaries from advanced MLLMs and their susceptibility to various bias.
BibTeX
@misc{liu2025mdseval,
  author = {Yinhong Liu and Jianfeng He and Hang Su and Ruixue Lian and Yi Nian and Jake W. Vincent and Srikanth Vishnubhotla and Robinson Piramuthu and Saab Mansour},
  title = {{MDSEval: A Meta-Evaluation Benchmark for Multimodal Dialogue Summarization}},
  year = {2025},
  eprint = {2510.01659},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.01659},
}
Figure