Large multimodal models evaluation: a survey

Zicheng Zhang, Junying Wang, Farong Wen, Yijin Guo, Xiangyu Zhao, Xinyu Fang, Shengyuan Ding, Ziheng Jia, Jiahao Xiao, Ye Shen, Yushuo Zheng, Xiaorong Zhu, Yalun Wu, Ziheng Jiao, Wei Sun, Zijian Chen, Kaiwei Zhang, Kang Fu, Yuqin Cao, Ming Hu, Yue Zhou 0005, Xuemei Zhou, Juntai Cao, Wei Zhou, Jinyu Cao, Ronghui Li, Donghao Zhou, Yuan Tian, Xiangyang Zhu, Chunyi Li, Haoning Wu, Xiaohong Liu, Junjun He, Yu Zhou 0016, Hui Liu, Lin Zhang, Zesheng Wang, Huiyu Duan, Yingjie Zhou, Xiongkuo Min, Qi Jia, Dongzhan Zhou, Wenlong Zhang, Jiezhang Cao, Xue Yang, Junzhi Yu, Songyang Zhang, Haodong Duan, Guangtao Zhai

Abstract

As large multimodal models (LMMs) advance rapidly across diverse multimodal understanding and generation tasks, the need for systematic and reliable evaluation frameworks becomes increasingly critical. To address this need, this survey provides a structured overview of LMM evaluation, centered around two main axes: multimodal evaluation for understanding and generation. (1) For understanding, a dual-perspective framework is introduced to distinguish benchmarks between general capabilities, which emphasize common tasks, and specialized capabilities, which reflect expert-level competence in domain-specific fields. (2) For generation, evaluation is organized by output modality, including image, video, audio, and 3D content. (3) From a community perspective, this survey further highlights authoritative leaderboards and foundational tools that have been instrumental in establishing a comprehensive evaluation ecosystem for LMMs. By unifying general-specialized understanding and modality-specific generation evaluations, this survey clarifies the current landscape and provides guidance for future research in the LMM evaluation field.
BibTeX
@article{zhang2025large,
  author = {Zicheng Zhang and Junying Wang and Farong Wen and Yijin Guo and Xiangyu Zhao and Xinyu Fang and Shengyuan Ding and Ziheng Jia and Jiahao Xiao and Ye Shen and Yushuo Zheng and Xiaorong Zhu and Yalun Wu and Ziheng Jiao and Wei Sun and Zijian Chen and Kaiwei Zhang and Kang Fu and Yuqin Cao and Ming Hu and Yue Zhou and Xuemei Zhou and Juntai Cao and Wei Zhou and Jinyu Cao and Ronghui Li and Donghao Zhou and Yuan Tian and Xiangyang Zhu and Chunyi Li and Haoning Wu and Xiaohong Liu and Junjun He and Yu Zhou and Hui Liu and Lin Zhang and Zesheng Wang and Huiyu Duan and Yingjie Zhou and Xiongkuo Min and Qi Jia and Dongzhan Zhou and Wenlong Zhang and Jiezhang Cao and Xue Yang and Junzhi Yu and Songyang Zhang and Haodong Duan and Guangtao Zhai},
  title = {{Large multimodal models evaluation: a survey}},
  journal = {Science China Information Sciences},
  year = {2025},
  doi = {10.1007/s11432-025-4676-4},
}

Loading related content...

Figure