SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension

Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, Ying ShanarXiv 2023 · DOI · arXiv

Abstract

SEED-Bench is a comprehensive evaluation suite comprising 19,000 human-annotated multiple-choice questions designed to assess the generative comprehension of Multimodal Large Language Models (MLLMs). Spanning 12 spatial and temporal dimensions across images and video, SEED-Bench provides an objective, reproducible leaderboard. Initial results indicate that while current models excel in basic identification, they struggle significantly with temporal dynamics and fine-grained spatial reasoning.

Introduction

The rapid advancement of Multimodal Large Language Models (MLLMs), such as LLaVA and MiniGPT-4, has outpaced the development of robust evaluation frameworks. Existing benchmarks often suffer from limited scale (fewer than 3,000 samples), subjective evaluation metrics (relying on "LLM-as-a-judge" scoring), and a narrow focus on static images that ignores temporal video dynamics. To reach Artificial General Intelligence, models must unify comprehension and generation. SEED-Bench addresses the comprehension gap by providing a standardized, large-scale, and objective yardstick for the community.

Methodology

The researchers developed an advanced pipeline to generate high-quality multiple-choice questions (MCQs) efficiently:

  1. Visual Information Extraction: For images, a "committee" of foundation models (BLIP-2 for captions, SAM for segmentation, and PaddleOCR for text) converted visual data into rich text descriptions. For videos, the authors utilized existing high-quality human annotations.
  2. GPT-4 Generation: These descriptions were fed into GPT-4 to generate MCQs featuring one correct answer and three plausible distractors.
  3. Visual Necessity Filtering: To ensure questions required actual visual input, they were presented to text-only LLMs (e.g., LLaMA, Vicuna). If a model could guess the correct answer using language priors alone, the question was discarded.
  4. Human Verification: Human annotators verified every question, selected the correct option, and categorized them into 12 distinct dimensions.

To evaluate models objectively, SEED-Bench uses a log-likelihood approach. Rather than testing instruction-following via free-text generation, the benchmark calculates which of the four option strings the model finds most likely given the visual context.

The SEED-Bench Dataset

SEED-Bench consists of 19,242 MCQs, making it approximately six times larger than MMBench. The data is sourced from CC3M for images and SSV2, Epic-Kitchens 100, and the Breakfast dataset for video.

Figure 1: Distribution of SEED-Bench evaluation dimensions and sample counts across spatial and temporal categories.

The benchmark evaluates models across 12 dimensions: nine spatial (e.g., Scene Understanding, Instance Location, Counting) and three temporal (Action Recognition, Action Prediction, and Procedure Understanding).

Results and Key Insights

The authors benchmarked 18 models, including pure LLMs, ImageLLMs, and VideoLLMs.

Figure 2: Leaderboard of MLLM performance on SEED-Bench, highlighting the top-performing models and the 50% accuracy barrier.
  • The Top Performer: InstructBLIP (Vicuna-7B) achieved the highest overall accuracy at 53.37%. Its success is attributed to a massive instruction-tuning set of 16 million samples, which includes specific OCR and temporal reasoning data.
  • The 50% Barrier: Most current MLLMs failed to reach 50% average accuracy, highlighting a significant gap between qualitative demonstrations and robust visual understanding.
  • Spatial vs. Temporal Reasoning: While models are proficient at identifying objects (Instance Identity), they perform poorly on spatial relations (identifying where an object is relative to others). Furthermore, models specifically designed for video (VideoLLMs) did not consistently outperform ImageLLMs on temporal tasks, suggesting that temporal architecture alone does not guarantee deep dynamic understanding.
  • Text Recognition: Performance in OCR-related tasks remained low (often <40%) for most models, with the exception of InstructBLIP.

Conclusion and Future Work

SEED-Bench reveals that current MLLMs still struggle with complex spatial reasoning and temporal dynamics. While the benchmark provides an objective scoring mechanism via MCQs, evaluating open-ended multimodal generation remains a challenge. Future work will focus on scaling the automated video annotation pipeline and maintaining a live leaderboard to track the rapid progress of the MLLM field.

Summary updated

Abstract

Based on powerful Large Language Models (LLMs), recent generative Multimodal Large Language Models (MLLMs) have gained prominence as a pivotal research area, exhibiting remarkable capability for both comprehension and generation. In this work, we address the evaluation of generative comprehension in MLLMs as a preliminary step towards a comprehensive assessment of generative models, by introducing a benchmark named SEED-Bench. SEED-Bench consists of 19K multiple choice questions with accurate human annotations (x 6 larger than existing benchmarks), which spans 12 evaluation dimensions including the comprehension of both the image and video modality. We develop an advanced pipeline for generating multiple-choice questions that target specific evaluation dimensions, integrating both automatic filtering and manual verification processes. Multiple-choice questions with groundtruth options derived from human annotation enables an objective and efficient assessment of model performance, eliminating the need for human or GPT intervention during evaluation. We further evaluate the performance of 18 models across all 12 dimensions, covering both the spatial and temporal understanding. By revealing the limitations of existing MLLMs through evaluation results, we aim for SEED-Bench to provide insights for motivating future research. We will launch and consistently maintain a leaderboard to provide a platform for the community to assess and investigate model capability.
BibTeX
@misc{li2023seed,
  author = {Bohao Li and Rui Wang and Guangzhi Wang and Yuying Ge and Yixiao Ge and Ying Shan},
  title = {{SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension}},
  year = {2023},
  eprint = {2307.16125},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.16125},
}
Figure