Abstract
SEED-Bench is a comprehensive evaluation suite comprising 19,000 human-annotated multiple-choice questions designed to assess the generative comprehension of Multimodal Large Language Models (MLLMs). Spanning 12 spatial and temporal dimensions across images and video, SEED-Bench provides an objective, reproducible leaderboard. Initial results indicate that while current models excel in basic identification, they struggle significantly with temporal dynamics and fine-grained spatial reasoning.
Introduction
The rapid advancement of Multimodal Large Language Models (MLLMs), such as LLaVA and MiniGPT-4, has outpaced the development of robust evaluation frameworks. Existing benchmarks often suffer from limited scale (fewer than 3,000 samples), subjective evaluation metrics (relying on "LLM-as-a-judge" scoring), and a narrow focus on static images that ignores temporal video dynamics. To reach Artificial General Intelligence, models must unify comprehension and generation. SEED-Bench addresses the comprehension gap by providing a standardized, large-scale, and objective yardstick for the community.
Methodology
The researchers developed an advanced pipeline to generate high-quality multiple-choice questions (MCQs) efficiently:
- Visual Information Extraction: For images, a "committee" of foundation models (BLIP-2 for captions, SAM for segmentation, and PaddleOCR for text) converted visual data into rich text descriptions. For videos, the authors utilized existing high-quality human annotations.
- GPT-4 Generation: These descriptions were fed into GPT-4 to generate MCQs featuring one correct answer and three plausible distractors.
- Visual Necessity Filtering: To ensure questions required actual visual input, they were presented to text-only LLMs (e.g., LLaMA, Vicuna). If a model could guess the correct answer using language priors alone, the question was discarded.
- Human Verification: Human annotators verified every question, selected the correct option, and categorized them into 12 distinct dimensions.
To evaluate models objectively, SEED-Bench uses a log-likelihood approach. Rather than testing instruction-following via free-text generation, the benchmark calculates which of the four option strings the model finds most likely given the visual context.
The SEED-Bench Dataset
SEED-Bench consists of 19,242 MCQs, making it approximately six times larger than MMBench. The data is sourced from CC3M for images and SSV2, Epic-Kitchens 100, and the Breakfast dataset for video.
The benchmark evaluates models across 12 dimensions: nine spatial (e.g., Scene Understanding, Instance Location, Counting) and three temporal (Action Recognition, Action Prediction, and Procedure Understanding).
Results and Key Insights
The authors benchmarked 18 models, including pure LLMs, ImageLLMs, and VideoLLMs.
Performance Trends
- The Top Performer: InstructBLIP (Vicuna-7B) achieved the highest overall accuracy at 53.37%. Its success is attributed to a massive instruction-tuning set of 16 million samples, which includes specific OCR and temporal reasoning data.
- The 50% Barrier: Most current MLLMs failed to reach 50% average accuracy, highlighting a significant gap between qualitative demonstrations and robust visual understanding.
- Spatial vs. Temporal Reasoning: While models are proficient at identifying objects (Instance Identity), they perform poorly on spatial relations (identifying where an object is relative to others). Furthermore, models specifically designed for video (VideoLLMs) did not consistently outperform ImageLLMs on temporal tasks, suggesting that temporal architecture alone does not guarantee deep dynamic understanding.
- Text Recognition: Performance in OCR-related tasks remained low (often <40%) for most models, with the exception of InstructBLIP.
Conclusion and Future Work
SEED-Bench reveals that current MLLMs still struggle with complex spatial reasoning and temporal dynamics. While the benchmark provides an objective scoring mechanism via MCQs, evaluating open-ended multimodal generation remains a challenge. Future work will focus on scaling the automated video annotation pipeline and maintaining a live leaderboard to track the rapid progress of the MLLM field.
Summary updated