Automatic, Meta and Human Evaluation for Multimodal Summarization with Multimodal Output

Haojie Zhuang, Wei Emma Zhang, Leon Xie, Weitong Chen, Jian Yang, Quan Sheng

Introduction

Multimodal Summarization with Multimodal Output (MSMO) is the task of condensing large collections of text and images into a concise text summary paired with the most relevant visual content. Despite the growth of this field, evaluation remains a significant challenge. Traditional metrics like ROUGE are designed for text and require expensive human-written references. Furthermore, human evaluation—often considered the gold standard—is poorly understood in a multimodal context and prone to cognitive biases. This paper introduces a comprehensive framework to standardize MSMO evaluation through automatic metrics, meta-evaluation benchmarks, and an analysis of human judgment.

A Unified Evaluation Framework

The authors propose a three-pronged research framework to address the limitations of current MSMO assessment methods. As shown in Figure 1, this framework encompasses automatic evaluation (RQ1), meta-evaluation to measure metric alignment with human intuition (RQ2), and a systematic study of human cognitive biases (RQ3).

The proposed evaluation framework for MSMO, illustrating the relationships between automatic metrics, meta-evaluation, and human annotation.

mLLM-EVAL: Reference-Free Automatic Evaluation

A key contribution of this work is mLLM-EVAL, a reference-free metric leveraging multimodal Large Language Models (such as GPT-4V). Unlike traditional metrics that compare a candidate to a human-written "gold standard," mLLM-EVAL evaluates the summary directly against the source document and images. By using specific prompts to score attributes like text relevance, coherence, image quality, and the "text-image" fit, it captures quality more holistically and avoids the subjectivity inherent in single human references.

Identifying Human Cognitive Biases

The study reveals that human evaluation is not a perfect baseline due to three critical cognitive biases:

  1. Anchoring Bias: Annotators become significantly harsher when shown a "perfect" reference summary alongside the AI-generated candidate.
  2. Lead Bias: Humans naturally prefer the first few images in a source set, regardless of their actual relevance to the summary content.
  3. Text Bias (The Halo Effect): A high-quality text summary can "trick" humans into giving the associated images a higher score than they deserve.

Figure 2 demonstrates the impact of anchoring bias, where the presence of references consistently leads to lower human scores across multiple quality dimensions such as relevance (rel), coherence (coh), and fluency (flu).

Bar charts illustrating the impact of anchoring bias; human scores are notably lower when reference summaries are provided during evaluation.

Key Results and Benchmark

The authors contributed a new meta-evaluation benchmark consisting of 1,562 human-annotated examples, providing a ground truth for testing future metrics. Key findings include:

  • Superior Correlation: mLLM-EVAL achieved a Pearson correlation of 0.63 for overall quality, significantly outperforming previous state-of-the-art metrics like MMAE (0.55).
  • Effective Image Selection: The metric proved better than traditional Image Precision at identifying which images truly "belong" with a specific text summary.
  • Scalability: The high correlation with human judgment suggests that multimodal LLMs can serve as a scalable, cost-effective alternative to human panels for routine model development.

Conclusion and Future Work

By identifying human biases and introducing a robust, reference-free metric, this work provides a foundation for more objective MSMO research. Future directions include expanding the framework to include video and audio modalities and testing the metrics across different languages and cultures. While proprietary LLM costs remain a factor, the shift toward reference-free evaluation marks a significant step toward more reliable multimodal AI systems.

Summary updated

Abstract

Multimodal summarization with multimodal output (MSMO) has attracted increasing research interests recently as multimodal summary could provide more comprehensive information compared to text-only summary, effectively improving the user experience and satisfaction. As one of the most fundamental components for the development of MSMO, evaluation is an emerging yet underexplored research topic. In this paper, we fill this gap and propose a research framework that studies three research questions of MSMO evaluation: (1) Automatic Evaluation: We propose a novel metric mLLM-EVAL, which utilizes multimodal Large Language Model for MSMO EVALuation. (2) Meta-Evaluation: We create a meta-evaluation benchmark dataset by collecting human-annotated scores for multimodal summaries. With our benchmark, we conduct meta-evaluation analysis to assess the quality of different evaluation metrics and show the effectiveness of our proposed mLLM-EVAL. (3) Human Evaluation: To provide more objective and unbiased human annotations for meta-evaluation, we hypothesize and verify three types of cognitive biases in human evaluation. We also incorporate our findings into the human annotation process in the meta-evaluation benchmark. Overall, our research framework provides an evaluation metric, a meta-evaluation benchmark dataset annotated by humans and an analysis of cognitive biases in human evaluation, which we believe would serve as a valuable and comprehensive resource for the MSMO research community.
BibTeX
@inproceedings{zhuang2024automatic,
  author = {Haojie Zhuang and Wei Emma Zhang and Leon Xie and Weitong Chen and Jian Yang and Quan Sheng},
  title = {{Automatic, Meta and Human Evaluation for Multimodal Summarization with Multimodal Output}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.430},
}
Figure