Towards Perceiving Small Visual Details in Zero-shot Visual Question Answering with Multimodal LLMs

Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, Filip Ilievski

TL;DR

Multimodal Large Language Models (MLLMs) suffer from a significant "perception gap" when dealing with small visual details, with zero-shot accuracy dropping by up to 46% as object size decreases. This paper demonstrates that this failure is causal—simply "zooming in" via cropping fixes it—and introduces five automated cropping methods that boost performance by leveraging either external models or the MLLM’s own internal attention and gradients.

What Problem Does This Paper Address?

While MLLMs like BLIP-2 and GPT-4 have shown impressive zero-shot capabilities, they often stumble on questions requiring fine-grained visual perception, such as reading a small street sign or identifying a tiny bird species.

The authors set out to answer two questions:

  1. How sensitive are these models to the physical size of the subject in a question?
  2. Is the failure due to a lack of reasoning/knowledge or a fundamental limitation in visual perception?

By partitioning datasets based on object size, they found a massive performance cliff. However, they also discovered that if a human manually crops the image to the relevant area, the model’s accuracy skyrockets. This suggests the models have the "intelligence" to answer the question but lack the "eyesight" to resolve small details at standard input resolutions.

Figure 2: Log-probability of correct answers increases as the image is cropped closer to the subject, demonstrating the causal link between visual scale and model performance across different tasks.

Methods

The researchers propose five automatic visual cropping methods to find the "Region of Interest" (RoI) at inference time:

External Knowledge Methods

  • clip-CROP: Iteratively crops the image from four sides, using CLIP to select the crop with the highest semantic similarity to the question.
  • yolo-CROP: Uses a YOLOv8 object detector to find candidate objects and selects the one most relevant to the question via CLIP.
  • sam-CROP: Uses the Segment Anything Model (SAM) to generate granular masks, converting them to bounding boxes for selection.

Native Visual Cropping (The "Aha!" Approach)

These methods are particularly novel because they don't require extra models; they look at the MLLM’s own "thought process":

  • grad-CROP: Tracks the gradient of the model’s decision back to the input pixels to see which area "triggered" the answer.
  • att-CROP: Aggregates attention scores from the model’s Transformer layers (specifically the Q-Former and LLM) to identify where the model is internally focusing.
Figure 5: Overview of the visual cropping framework, showing how different cropping strategies (clip, yolo, att, sam) are integrated with MLLM architectures like BLIP-2.

Key Insights and Ideas

  • The Perception Bottleneck is Causal: The fact that cropping (without changing the question or the model) restores accuracy proves that the limitation is in the visual front-end, not the linguistic reasoning.
  • Models Know "Where," but not "What": The success of att-CROP shows that even when a model gives the wrong answer, its internal attention is often already focused on the correct region. It just can't "see" clearly enough to identify the object until it is zoomed in.
  • The Global-Local Trade-off: Cropping is not a silver bullet. While it helps with "What color is the bird?", it can hurt "How many birds are there?" because the model loses the global context. The authors suggest a "question-gate" to decide when to crop based on the question type.

Key Results

  • Massive Accuracy Gains: On the TextVQA dataset, human-level cropping improved BLIP-2 FlanT5 accuracy from 25.91% to 37.68%.
  • Native Methods Compete: The att-CROP method (using the model's own attention) achieved 34.26% on TextVQA, significantly outperforming the baseline without needing an external detector like YOLO.
  • FDVQA Success: On the authors' new fine-detail dataset (FDVQA), clip-CROP improved accuracy from 33.94% to 36.61%, while human cropping reached 42.29%.

Limitations and Future Work

  • The "Why" Remains Elusive: While the paper proves the perception limit exists, it doesn't fully explain why architectures with high-resolution encoders (like ViT) still fail to resolve these details in a zero-shot setting.
  • Fusing Scales: The current approach simply concatenates original and cropped tokens. The authors note that more sophisticated multi-scale fusion mechanisms are needed to maintain global context while perceiving local details.

Summary updated

Abstract

Multimodal Large Language Models (MLLMs) have recently achieved promising zero-shot accuracy on visual question answering (VQA) -- a fundamental task affecting various downstream applications and domains. Given the great potential for the broad use of these models, it is important to investigate their limitations in dealing with different image and question properties. In this work, we investigate whether MLLMs can perceive small details as well as large details in images. In particular, we show that their zero-shot accuracy in answering visual questions is very sensitive to the size of the visual subject of the question, declining up to 46% with size. Furthermore, we show that this effect is causal by observing that human visual cropping can significantly mitigate their sensitivity to size. Inspired by the usefulness of human cropping, we then propose five automatic visual cropping methods -- leveraging either external localization models or the decision process of the given MLLM itself -- as inference time mechanisms to improve the zero-shot performance of MLLMs. We study their effectiveness on four popular VQA datasets, and a subset of the VQAv2 dataset tailored towards fine visual details. Our findings suggest that MLLMs should be used with caution in detail-sensitive VQA applications, and that visual cropping is a promising direction to improve their zero-shot performance. To facilitate further investigation of MLLMs' behaviors, our code and data are publicly released.
BibTeX
@inproceedings{zhang2023towards,
  author = {Jiarui Zhang and Mahyar Khayatkhoei and Prateek Chhikara and Filip Ilievski},
  title = {{Towards Perceiving Small Visual Details in Zero-shot Visual Question Answering with Multimodal LLMs}},
  booktitle = {R0-FoMo @ NeurIPS},
  year = {2023},
  url = {https://openreview.net/forum?id=YrYcoV2dAk},
}
Figure