Introduction
Multimodal Summarization with Multimodal Output (MSMO) is the task of condensing large collections of text and images into a concise text summary paired with the most relevant visual content. Despite the growth of this field, evaluation remains a significant challenge. Traditional metrics like ROUGE are designed for text and require expensive human-written references. Furthermore, human evaluation—often considered the gold standard—is poorly understood in a multimodal context and prone to cognitive biases. This paper introduces a comprehensive framework to standardize MSMO evaluation through automatic metrics, meta-evaluation benchmarks, and an analysis of human judgment.
A Unified Evaluation Framework
The authors propose a three-pronged research framework to address the limitations of current MSMO assessment methods. As shown in Figure 1, this framework encompasses automatic evaluation (RQ1), meta-evaluation to measure metric alignment with human intuition (RQ2), and a systematic study of human cognitive biases (RQ3).
mLLM-EVAL: Reference-Free Automatic Evaluation
A key contribution of this work is mLLM-EVAL, a reference-free metric leveraging multimodal Large Language Models (such as GPT-4V). Unlike traditional metrics that compare a candidate to a human-written "gold standard," mLLM-EVAL evaluates the summary directly against the source document and images. By using specific prompts to score attributes like text relevance, coherence, image quality, and the "text-image" fit, it captures quality more holistically and avoids the subjectivity inherent in single human references.
Identifying Human Cognitive Biases
The study reveals that human evaluation is not a perfect baseline due to three critical cognitive biases:
- Anchoring Bias: Annotators become significantly harsher when shown a "perfect" reference summary alongside the AI-generated candidate.
- Lead Bias: Humans naturally prefer the first few images in a source set, regardless of their actual relevance to the summary content.
- Text Bias (The Halo Effect): A high-quality text summary can "trick" humans into giving the associated images a higher score than they deserve.
Figure 2 demonstrates the impact of anchoring bias, where the presence of references consistently leads to lower human scores across multiple quality dimensions such as relevance (rel), coherence (coh), and fluency (flu).
Key Results and Benchmark
The authors contributed a new meta-evaluation benchmark consisting of 1,562 human-annotated examples, providing a ground truth for testing future metrics. Key findings include:
- Superior Correlation: mLLM-EVAL achieved a Pearson correlation of 0.63 for overall quality, significantly outperforming previous state-of-the-art metrics like MMAE (0.55).
- Effective Image Selection: The metric proved better than traditional Image Precision at identifying which images truly "belong" with a specific text summary.
- Scalability: The high correlation with human judgment suggests that multimodal LLMs can serve as a scalable, cost-effective alternative to human panels for routine model development.
Conclusion and Future Work
By identifying human biases and introducing a robust, reference-free metric, this work provides a foundation for more objective MSMO research. Future directions include expanding the framework to include video and audio modalities and testing the metrics across different languages and cultures. While proprietary LLM costs remain a factor, the shift toward reference-free evaluation marks a significant step toward more reliable multimodal AI systems.
Summary updated