Visual Cropping Improves Zero-Shot Question Answering of Multimodal Large Language Models

Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, Filip Ilievski

TL;DR

Multimodal Large Language Models (MLLMs) struggle significantly with answering questions about small visual details, suffering performance drops of up to 46% as target objects shrink. The authors show this sensitivity is causal—zooming in via cropping largely restores accuracy—and propose three automated, inference-time cropping strategies that boost zero-shot VQA performance without retraining the underlying models.

What Problem Does This Paper Address?

While multimodal LLMs such as BLIP-2 achieve impressive zero-shot performance on Visual Question Answering (VQA), they frequently stumble when the question hinges on subtle visual details (e.g., recognizing a tiny street sign or distinguishing an egret from a generic white bird).

Because visual encoders compress whole images into fixed-size representations or token grids, small regions lose spatial resolution. This paper investigates whether MLLM errors on fine-grained visual questions stem from semantic misunderstanding or visual perception limitations, and whether automated visual cropping at inference time can resolve the bottleneck.

Methods

To evaluate whether size sensitivity is causal, the authors first evaluate human-CROP, which provides the model with crops around ground-truth bounding boxes. To translate this into an automated inference-time pipeline, they design three cropping strategies:

  1. clip-CROP: Progressively crops the image in multiple steps toward the sub-region with the highest CLIP text-image similarity to the question.
  2. yolo-CROP: Uses YOLOv8 object proposals as candidate bounding boxes to prevent CLIP from fixating on empty or irrelevant background patches.
  3. sam-CROP: Uses the Segment Anything Model (SAM) to generate category-agnostic segmentation masks across the image, scoring candidate regions with CLIP to select the crop.

To prevent cropping from discarding necessary global scene context, the authors extract visual tokens from both the original full image and the cropped image, concatenating them before passing them to the language model.

Original Image  ───► Vision Encoder ──┐
                                      ├──► [Image Tokens || Crop Tokens] ──► LLM ──► Answer
Cropped Region  ───► Vision Encoder ──┘

Datasets

The authors evaluate models on five benchmarks:

  • TextVQA: Emphasizes fine details through OCR text reading, where over half the target text occupies less than 0.5% of the image area.
  • FDVQA (Fine-Detail VQA): A new benchmark introduced by the authors comprising 109 challenging questions from VQAv2 where BLIP-2 originally failed, verified by three human annotators to strictly require resolving small visual details, annotated with ground-truth bounding boxes.
  • VQAv2, GQA, and A-OKVQA: Standard, diverse VQA benchmarks covering compositional reasoning, knowledge-based reasoning, and general perception.

Key Insights and Ideas

  • The bottleneck is visual perception, not reasoning: When zoomed in, BLIP-2 shifts from generic predictions ("white bird") to fine-grained identifications ("egret") without altering the text prompt.
  • The decline is dramatic but reversible: In TextVQA, BLIP-2 (Flan-T5) accuracy drops by 46% between large targets ($S \ge 0.05$ of the image) and tiny targets ($S < 0.005$). Ground-truth cropping reduces this drop to under 19%.
  • Architecture matters for visual token concatenation: Visual cropping reliably benefits encoder-decoder models (BLIP-2 Flan-T5 across all benchmarks). However, for decoder-only models (BLIP-2 OPT), concatenating crop tokens can degrade overall performance on general VQA datasets, likely because decoder-only self-attention expects visual tokens at specific prompt-initial positions.
  • Task trade-offs: Cropping provides significant gains on attribute identification and text reading, but hurts questions that require holistic scene context, such as counting or spatial localization.

Key Results

  • Fine-Detail Benchmarks:
  • On FDVQA, base BLIP-2 Flan-T5 achieves 33.94% accuracy. clip-CROP raises this to 36.61% (while ground-truth human-CROP achieves 42.29%).
  • On TextVQA, Flan-T5 baseline accuracy of 25.91% improves to 30.93% with clip-CROP and 32.31% with sam-CROP (human-CROP reaches 37.68%).
  • General Benchmarks:
  • Flan-T5 shows modest improvements across broad benchmarks: GQA increases from 43.85% to 45.23% (sam-CROP), and A-OKVQA increases from 43.42% to 45.34% (clip-CROP).
  • Speed & Latency:
  • yolo-CROP is fastest (0.355s on an NVIDIA RTX A5000), but clip-CROP (1.072s) offers the best balance of speed and accuracy. sam-CROP is the heaviest (3.329s on GPU; >91s on CPU).

Limitations and Future Work

  • The Human-Machine Gap: While automated methods help, a notable performance gap remains between automated heuristic cropping and human-annotated cropping.
  • Global Context Loss: Merging crops via naive token concatenation harms tasks requiring holistic scene views (e.g., counting and spatial relations).
  • Future Directions: Developing dynamic, question-conditioned routing (deciding when to crop) and more sophisticated cross-attention fusion mechanisms to better blend local crops with global views.

Summary updated

Abstract

Multimodal Large Language Models (LLMs) have recently achieved promising zero-shot accuracy on visual question answering (VQA) -- a fundamental task affecting various downstream applications and domains. Given the great potential for the broad use of these models, it is important to investigate their limitations in dealing with different image and question properties. In this work, we investigate whether multimodal LLMs can perceive small details as well as large details in images. In particular, we show that their zero-shot accuracy in answering visual questions is very sensitive to the size of the visual subject of the question, declining up to $46\%$ with size. Furthermore, we show that this effect is causal by observing that human visual cropping can significantly mitigate their sensitivity to size. Inspired by the usefulness of human cropping, we then propose three automatic visual cropping methods as inference time mechanisms to improve the zero-shot performance of multimodal LLMs. We study their effectiveness on four popular VQA datasets, and a subset of the VQAv2 dataset tailored towards fine visual details. Our findings suggest that multimodal LLMs should be used with caution in detail-sensitive VQA applications, and that visual cropping is a promising direction to improve their zero-shot performance.
BibTeX
@inproceedings{zhang2023visual,
  author = {Jiarui Zhang and Mahyar Khayatkhoei and Prateek Chhikara and Filip Ilievski},
  title = {{Visual Cropping Improves Zero-Shot Question Answering of Multimodal Large Language Models}},
  booktitle = {R0-FoMo @ NeurIPS},
  year = {2023},
  url = {https://openreview.net/forum?id=YrYcoV2dAk},
}

Loading related content...

Figure