Abstract: Yinhong Liu, Jianfeng He, Hang Su, Ruixue Lian, Yi Nian, Jake W. Vincent, Srikanth Vishnubhotla, Robinson Piramuthu, Saab Mansour. Findings of the Association for Computational Linguistics: EMNLP 2025.
2025.
Introduction
As artificial intelligence evolves toward seamless multimodal interaction, Multimodal Dialogue Summarization (MDS)—the task of condensing conversations involving both text and visual elements—has become increasingly critical. However, evaluating the quality of these summaries remains a significant challenge. While human evaluation is the gold standard, it is prohibitively expensive and slow. Conversely, the reliability of "LLM-as-a-judge" metrics for this complex task has not been rigorously tested.
Furthermore, existing MDS datasets often feature high information redundancy, where images and text convey identical information. This allows models to succeed without genuine multimodal reasoning. To address these gaps, the authors present MDSEval, a comprehensive meta-evaluation benchmark designed to assess the alignment between automatic evaluators and human experts.
Methodology and Dataset Curation
The development of MDSEval follows a rigorous multi-stage pipeline designed to ensure high-quality, non-redundant data and diverse summary candidates.
Mutually Exclusive Key Information (MEKI)
The core technical innovation of this work is the MEKI filtering metric. MEKI quantifies the "exclusive" information provided by one modality relative to another using CLIP embeddings. By calculating the orthogonal component of an image embedding ($I$) relative to the text embedding ($T$), the authors can select dialogues where the visual component provides unique, complementary information essential for a complete summary. This ensures that the benchmark truly tests multimodal reasoning rather than simple text extraction.
Summary Generation and Expert Annotation
To test evaluator robustness, the benchmark includes summaries generated by four state-of-the-art models (Molmo-72B, GPT-4o-mini, Gemini 1.5 Flash, and Qwen-VL-Max) using various prompting strategies, including Zero-shot and In-Context Learning. This resulted in a dataset of 198 dialogues and 990 summaries. Each summary was evaluated by three human experts across eight dimensions, including Coherence, Multimodal Information Balancing, and Topic Progression, providing nearly 3,000 ground-truth annotations.
Key Insights and Results
The meta-evaluation revealed several critical weaknesses in current Multimodal Large Language Model (MLLM) evaluators. While models like GPT-4o and Gemini 1.5 Flash are capable of generating summaries, their ability to judge them is limited. Spearman correlation between model-assigned scores and human judgments was strikingly low, often falling below 0.10 for dimensions like Coherence.
One prominent issue is the "Hedge" Bias: unlike human experts who utilize the full 1-5 rating scale, MLLM evaluators tend to cluster their scores around "4." This lack of variance prevents them from distinguishing between "good" and "excellent" summaries, making them poor tools for fine-grained benchmarking. Furthermore, models exhibited significant Positional Bias in pairwise comparisons; for instance, Gemini-1.5-flash showed a consistent preference for the first option presented.
The authors also identified a Translation Loss in "Image-prompting" methods. When images are converted into text descriptions before evaluation, crucial visual nuances are lost, leading to poor scores for visual information coverage. Finally, a significant Faithfulness Deficit was observed. All tested models struggled to detect "fine-grained unfaithfulness"—hallucinations where a summary describes visual elements not present in the source image. Balanced accuracy in faithfulness detection fell below 30%, suggesting that current AI cannot yet reliably verify the factual accuracy of multimodal summaries.
Conclusion
MDSEval highlights a sobering reality: current MLLMs are not yet reliable judges for multimodal dialogue summarization. The benchmark serves as a vital tool for the community to track progress in multimodal reasoning and evaluation. Future work will expand the benchmark to include video and audio modalities and explore more diverse domains beyond social chitchat.
Summary updated