Visual Cropping Improves Zero-Shot Question Answering of Multimodal Large Language Models

Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, Filip Ilievski

Abstract

Multimodal Large Language Models (MLLMs) have recently achieved promising zero-shot accuracy on visual question answering (VQA) -- a fundamental task affecting various downstream applications and domains. Given the great potential for the broad use of these models, it is important to investigate their limitations in dealing with different image and question properties. In this work, we investigate whether MLLMs can perceive small details as well as large details in images. In particular, we show that their zero-shot accuracy in answering visual questions is very sensitive to the size of the visual subject of the question, declining up to 46% with size. Furthermore, we show that this effect is causal by observing that human visual cropping can significantly mitigate their sensitivity to size. Inspired by the usefulness of human cropping, we then propose five automatic visual cropping methods -- leveraging either external localization models or the decision process of the given MLLM itself -- as inference time mechanisms to improve the zero-shot performance of MLLMs. We study their effectiveness on four popular VQA datasets, and a subset of the VQAv2 dataset tailored towards fine visual details. Our findings suggest that MLLMs should be used with caution in detail-sensitive VQA applications, and that visual cropping is a promising direction to improve their zero-shot performance.

Introduction: The Perceptual "Blind Spot" of MLLMs

While Multimodal Large Language Models (MLLMs) have demonstrated impressive zero-shot capabilities in Visual Question Answering (VQA), they exhibit a significant literal blind spot: small visual details. Current research indicates that while a model may perform well when asked about large objects, its performance craters when tasked with reading small street signs or identifying specific bird species in the distance.

This study investigates whether these failures are semantic (a lack of question understanding) or perceptual (an inability to resolve pixels clearly). By demonstrating that human-guided "zooming" (cropping) resolves these errors, the authors establish that the issue is primarily perceptual. This suggests that MLLMs often possess the internal "logic" to find an answer but lack the "acuity" to process it at standard input resolutions (typically 224x224 or 336x336).

Methodology: Automatic Visual Cropping

To address this perception gap, five automatic visual cropping methods were developed to identify the "Region of Interest" (ROI) at inference time. These methods are categorized into external knowledge-based approaches and native approaches that leverage the MLLM's internal states.

External Knowledge Methods

These methods utilize specialized computer vision models to identify the ROI:

  • clip-CROP: Iteratively crops the image from four sides, using CLIP to select the crop with the highest semantic similarity to the question.
  • yolo-CROP: Employs a YOLOv8 object detector to find candidate objects, followed by CLIP-based selection.
  • sam-CROP: Uses the Segment Anything Model (SAM) to generate granular masks, which are converted to bounding boxes for selection.

Native Visual Cropping

These novel methods use the MLLM’s own internal decision-making process to determine where to zoom:

  • grad-CROP: Tracks the gradient of the model’s decision back to the input pixels to identify influential areas.
  • att-CROP: Aggregates attention scores from the model's Transformer blocks (specifically the Q-Former in BLIP-2) to identify where the model is "looking."
Overview of the visual cropping methods and the integration into the BLIP-2 architecture.

To maintain a balance between local detail and global context, the authors concatenated tokens from the original image with tokens from the high-resolution crop before passing them to the LLM decoder.

Experimental Results

The study evaluated these methods across several benchmarks, including TextVQA (focused on fine-detail text reading) and FDVQA (Fine Detail VQA), a new dataset of 109 "hard" samples from VQAv2 where BLIP-2 initially failed due to small details.

Size Sensitivity and Accuracy Gains

Without cropping, accuracy on TextVQA dropped from 36.8% for "large" subjects to just 19.9% for "small" subjects. Human-guided cropping improved BLIP-2's accuracy on TextVQA by a massive 45.35%. The best automatic method, att-CROP, successfully recovered approximately 71% of that potential gain.

Probability shifts for correct answers as crop size decreases across different visual tasks.

As shown in the figure above, decreasing the crop size (zooming in) consistently leads to a dramatic increase in the log-probability of the correct answer. This confirms that the model's internal representations are sensitive to the resolution of the subject matter.

Efficiency and Consistency

Native methods like att-CROP proved remarkably efficient, with inference times comparable to optimized object detectors like YOLO. Furthermore, att-CROP showed consistent improvements across almost all datasets, suggesting it is a robust general-purpose enhancement for zero-shot VQA.

Discussion and Limitations

While visual cropping significantly improves performance on detail-oriented questions, it introduces a "global context" trade-off. Zooming in can mislead models on questions requiring a wide view, such as counting objects or global localization. The authors suggest a "question-gate" mechanism to decide when to apply cropping, which remains a key area for future refinement.

Additionally, while the study focused on BLIP-2, the findings suggest that resolution-based perception remains a fundamental hurdle for billion-parameter models trained on massive datasets. Future work will explore how these findings generalize across diverse decoder-only MLLM architectures.

Summary updated

Abstract

Multimodal Large Language Models (MLLMs) have recently achieved promising zero-shot accuracy on visual question answering (VQA) -- a fundamental task affecting various downstream applications and domains. Given the great potential for the broad use of these models, it is important to investigate their limitations in dealing with different image and question properties. In this work, we investigate whether MLLMs can perceive small details as well as large details in images. In particular, we show that their zero-shot accuracy in answering visual questions is very sensitive to the size of the visual subject of the question, declining up to 46% with size. Furthermore, we show that this effect is causal by observing that human visual cropping can significantly mitigate their sensitivity to size. Inspired by the usefulness of human cropping, we then propose five automatic visual cropping methods -- leveraging either external localization models or the decision process of the given MLLM itself -- as inference time mechanisms to improve the zero-shot performance of MLLMs. We study their effectiveness on four popular VQA datasets, and a subset of the VQAv2 dataset tailored towards fine visual details. Our findings suggest that MLLMs should be used with caution in detail-sensitive VQA applications, and that visual cropping is a promising direction to improve their zero-shot performance. To facilitate further investigation of MLLMs' behaviors, our code and data are publicly released.
BibTeX
@inproceedings{zhang2023visual,
  author = {Jiarui Zhang and Mahyar Khayatkhoei and Prateek Chhikara and Filip Ilievski},
  title = {{Visual Cropping Improves Zero-Shot Question Answering of Multimodal Large Language Models}},
  booktitle = {R0-FoMo @ NeurIPS},
  year = {2023},
  url = {https://openreview.net/forum?id=YrYcoV2dAk},
}
Figure