TL;DR
This paper introduces MultipanelVQA, a new benchmark designed to test how well multimodal large language models (MLLMs) can understand images composed of multiple subfigures. They found that even state-of-the-art MLLMs struggle significantly with these complex layouts, performing far below human accuracy, highlighting a critical gap in current AI capabilities.
What Problem Does This Paper Address?
As AI systems become more sophisticated, they need to understand increasingly complex visual information. Multipanel images, like web screenshots or posters, are ubiquitous and convey information through the arrangement of multiple distinct subfigures. Current benchmarks often focus on single images, leaving a gap in evaluating MLLMs' ability to reason about the content, layout, and relationships within these multipanel compositions. This paper aims to bridge that gap by providing a challenging evaluation specifically for multipanel visual question answering (VQA).
Methods
The authors introduce the MultipanelVQA benchmark, a dataset of 6,600 question-answer triplets paired with multipanel images. The benchmark includes:
- Three distinct question types:
- Identifying common or unique content across subfigures.
- Pinpointing content within specific subfigures using positional descriptions.
- Locating subfigures via visual grounding in a multiple-choice format.
- A novel synthetic data generation process: This allows for controlled experiments by systematically varying factors like subfigure similarity, size, layout style, and the presence of visual text hints. This synthetic data is crucial for isolating the impact of specific image attributes on MLLM performance.
- Real-world data: Sourced from web screenshots and posters to ensure ecological validity.
- Ablation studies: To systematically analyze the impact of various visual elements (e.g., neighboring subfigures, background, visual text) on MLLM performance.
Datasets
- MultipanelVQA Benchmark: A newly introduced dataset comprising 6,600 image-question-answer triplets. It consists of two subsets:
- Real-world data: 100 multipanel images sourced from web screenshots and posters.
- Synthetic data: 2,100 artificially generated multipanel images designed for controlled analysis of various factors.
- Source Datasets for Synthetic Data: MagicBrush and VQAv2 were used as sources for individual subfigures.
Key Insights and Ideas
- MLLMs struggle with multipanel images: Despite impressive performance on single-image tasks, current MLLMs exhibit a significant performance drop when faced with multipanel images, highlighting a critical area for improvement.
- Layout matters: The arrangement of subfigures, not just their content, significantly impacts MLLM performance. Models tend to perform better with fewer subfigures and simpler layouts.
- Neighboring subfigures are a major distraction: Removing surrounding subfigures dramatically improves model performance, indicating that interference from adjacent elements is a primary challenge.
- Visual text prompts can help, but inconsistently: Adding sequential numbers to subfigure captions can benefit some models, especially when the question explicitly references these captions, suggesting a nuanced interaction between visual prompts and model architecture.
- Synthetic data is key for analysis: The ability to generate synthetic data with controlled variations allows for precise isolation and understanding of factors affecting MLLM performance, which is difficult with real-world data alone.
Key Results
- Human performance is near-perfect: Humans achieve approximately 99% accuracy on the MultipanelVQA benchmark.
- MLLM performance lags significantly: Even the best-performing proprietary models (GPT-4V, GPT-4o, Gemini Pro Vision) show a substantial performance drop compared to humans, with some open-source models performing close to random chance. For example, GPT-4o achieves around 75.5% average accuracy on synthetic data, while humans achieve 99%.
- Ablation studies reveal key challenges:
- Removing neighboring subfigures improved performance for all models, with some seeing gains of over 10% (e.g., LLaVA-NeXT).
- Reducing visual similarity among subfigures also led to performance improvements for most models.
- Increasing subfigure size generally improved performance, while increasing the number of subfigures consistently degraded performance across all models.
- Visual text hints have mixed effects: While removing backgrounds improved performance for top models, the impact of visual text hints (like captions) was varied, suggesting different sensitivities among MLLMs.
Limitations and Future Work
- GPT-4 as evaluator: The use of GPT-4 as an evaluator necessitated simplifying questions to yes/no or short-answer formats, limiting the depth of evaluation. Future work could explore more complex question types.
- Extreme underperformance of some models: The very low accuracy of some models on the synthetic data makes it difficult to derive meaningful conclusions from their error analysis.
- Future directions: The authors suggest further exploration of tailored strategies for integrating visual prompts into different MLLMs and developing more sophisticated algorithms for multipanel image understanding. The benchmark itself can also serve as a tool for selecting strong MLLMs for tasks like GUI understanding and inspire applications that present sequences of images as subfigures.
Summary updated