Muffin or Chihuahua? Challenging Large Vision-Language Models with Multipanel VQA

Yue Fan, Jing Gu, Kaiwen Zhou, Qianqi Yan, Shan Jiang, Ching-Chen Kuo, Xinze Guan, Xin Eric Wang

TL;DR

This paper introduces MultipanelVQA, a new benchmark designed to test how well multimodal large language models (MLLMs) can understand images composed of multiple subfigures. They found that even state-of-the-art MLLMs struggle significantly with these complex layouts, performing far below human accuracy, highlighting a critical gap in current AI capabilities.

What Problem Does This Paper Address?

As AI systems become more sophisticated, they need to understand increasingly complex visual information. Multipanel images, like web screenshots or posters, are ubiquitous and convey information through the arrangement of multiple distinct subfigures. Current benchmarks often focus on single images, leaving a gap in evaluating MLLMs' ability to reason about the content, layout, and relationships within these multipanel compositions. This paper aims to bridge that gap by providing a challenging evaluation specifically for multipanel visual question answering (VQA).

Methods

The authors introduce the MultipanelVQA benchmark, a dataset of 6,600 question-answer triplets paired with multipanel images. The benchmark includes:

  • Three distinct question types:
    1. Identifying common or unique content across subfigures.
  • Pinpointing content within specific subfigures using positional descriptions.
  • Locating subfigures via visual grounding in a multiple-choice format.
  • A novel synthetic data generation process: This allows for controlled experiments by systematically varying factors like subfigure similarity, size, layout style, and the presence of visual text hints. This synthetic data is crucial for isolating the impact of specific image attributes on MLLM performance.
  • Real-world data: Sourced from web screenshots and posters to ensure ecological validity.
  • Ablation studies: To systematically analyze the impact of various visual elements (e.g., neighboring subfigures, background, visual text) on MLLM performance.

Datasets

  • MultipanelVQA Benchmark: A newly introduced dataset comprising 6,600 image-question-answer triplets. It consists of two subsets:
  • Real-world data: 100 multipanel images sourced from web screenshots and posters.
  • Synthetic data: 2,100 artificially generated multipanel images designed for controlled analysis of various factors.
  • Source Datasets for Synthetic Data: MagicBrush and VQAv2 were used as sources for individual subfigures.

Key Insights and Ideas

  • MLLMs struggle with multipanel images: Despite impressive performance on single-image tasks, current MLLMs exhibit a significant performance drop when faced with multipanel images, highlighting a critical area for improvement.
  • Layout matters: The arrangement of subfigures, not just their content, significantly impacts MLLM performance. Models tend to perform better with fewer subfigures and simpler layouts.
  • Neighboring subfigures are a major distraction: Removing surrounding subfigures dramatically improves model performance, indicating that interference from adjacent elements is a primary challenge.
  • Visual text prompts can help, but inconsistently: Adding sequential numbers to subfigure captions can benefit some models, especially when the question explicitly references these captions, suggesting a nuanced interaction between visual prompts and model architecture.
  • Synthetic data is key for analysis: The ability to generate synthetic data with controlled variations allows for precise isolation and understanding of factors affecting MLLM performance, which is difficult with real-world data alone.

Key Results

  • Human performance is near-perfect: Humans achieve approximately 99% accuracy on the MultipanelVQA benchmark.
  • MLLM performance lags significantly: Even the best-performing proprietary models (GPT-4V, GPT-4o, Gemini Pro Vision) show a substantial performance drop compared to humans, with some open-source models performing close to random chance. For example, GPT-4o achieves around 75.5% average accuracy on synthetic data, while humans achieve 99%.
  • Ablation studies reveal key challenges:
    • Removing neighboring subfigures improved performance for all models, with some seeing gains of over 10% (e.g., LLaVA-NeXT).
  • Reducing visual similarity among subfigures also led to performance improvements for most models.
  • Increasing subfigure size generally improved performance, while increasing the number of subfigures consistently degraded performance across all models.
  • Visual text hints have mixed effects: While removing backgrounds improved performance for top models, the impact of visual text hints (like captions) was varied, suggesting different sensitivities among MLLMs.

Limitations and Future Work

  • GPT-4 as evaluator: The use of GPT-4 as an evaluator necessitated simplifying questions to yes/no or short-answer formats, limiting the depth of evaluation. Future work could explore more complex question types.
  • Extreme underperformance of some models: The very low accuracy of some models on the synthetic data makes it difficult to derive meaningful conclusions from their error analysis.
  • Future directions: The authors suggest further exploration of tailored strategies for integrating visual prompts into different MLLMs and developing more sophisticated algorithms for multipanel image understanding. The benchmark itself can also serve as a tool for selecting strong MLLMs for tasks like GUI understanding and inspire applications that present sequences of images as subfigures.

Summary updated

Abstract

Multipanel images, commonly seen as web screenshots, posters, etc., pervade our daily lives. These images, characterized by their composition of multiple subfigures in distinct layouts, effectively convey information to people. Toward building advanced multimodal AI applications, such as agents that understand complex scenes and navigate through webpages, the skill of multipanel visual reasoning is essential, and a comprehensive evaluation of models in this regard is important. Therefore, we introduce Multipanel Visual Question Answering (MultipanelVQA), a novel benchmark comprising 6,600 triplets of questions, answers, and multipanel images that specifically challenge models in comprehending multipanel images. Our evaluation shows that questions in the MultipanelVQA benchmark pose significant challenges to the state-of-the-art Multimodal Large Language Models (MLLMs) tested, even though humans can attain approximately 99% accuracy on these questions. Distinctively, the MultipanelVQA benchmark features synthetically generated multipanel images specifically crafted to isolate and assess the impact of various factors, such as the layout, on MLLMs' multipanel image comprehension abilities. As a result, in addition to benchmarking the capabilities of MLLMs in understanding multipanel images, we analyze various factors of the multipanel image that affect MLLMs' performance with synthetic data and offer insights for enhancement. Code and data are released at https://sites.google.com/view/multipanelvqa/home.
BibTeX
@inproceedings{fan2024muffin,
  author = {Yue Fan and Jing Gu and Kaiwen Zhou and Qianqi Yan and Shan Jiang and Ching-Chen Kuo and Xinze Guan and Xin Eric Wang},
  title = {{Muffin or Chihuahua? Challenging Large Vision-Language Models with Multipanel VQA}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.370},
}
Figure