TL;DR
Multimodal Large Language Models (MLLMs) struggle significantly with answering questions about small visual details, suffering performance drops of up to 46% as target objects shrink. The authors show this sensitivity is causal—zooming in via cropping largely restores accuracy—and propose three automated, inference-time cropping strategies that boost zero-shot VQA performance without retraining the underlying models.
What Problem Does This Paper Address?
While multimodal LLMs such as BLIP-2 achieve impressive zero-shot performance on Visual Question Answering (VQA), they frequently stumble when the question hinges on subtle visual details (e.g., recognizing a tiny street sign or distinguishing an egret from a generic white bird).
Because visual encoders compress whole images into fixed-size representations or token grids, small regions lose spatial resolution. This paper investigates whether MLLM errors on fine-grained visual questions stem from semantic misunderstanding or visual perception limitations, and whether automated visual cropping at inference time can resolve the bottleneck.
Methods
To evaluate whether size sensitivity is causal, the authors first evaluate human-CROP, which provides the model with crops around ground-truth bounding boxes. To translate this into an automated inference-time pipeline, they design three cropping strategies:
clip-CROP: Progressively crops the image in multiple steps toward the sub-region with the highest CLIP text-image similarity to the question.yolo-CROP: Uses YOLOv8 object proposals as candidate bounding boxes to prevent CLIP from fixating on empty or irrelevant background patches.sam-CROP: Uses the Segment Anything Model (SAM) to generate category-agnostic segmentation masks across the image, scoring candidate regions with CLIP to select the crop.
To prevent cropping from discarding necessary global scene context, the authors extract visual tokens from both the original full image and the cropped image, concatenating them before passing them to the language model.
Original Image ───► Vision Encoder ──┐
├──► [Image Tokens || Crop Tokens] ──► LLM ──► Answer
Cropped Region ───► Vision Encoder ──┘
Datasets
The authors evaluate models on five benchmarks:
- TextVQA: Emphasizes fine details through OCR text reading, where over half the target text occupies less than 0.5% of the image area.
- FDVQA (Fine-Detail VQA): A new benchmark introduced by the authors comprising 109 challenging questions from VQAv2 where BLIP-2 originally failed, verified by three human annotators to strictly require resolving small visual details, annotated with ground-truth bounding boxes.
- VQAv2, GQA, and A-OKVQA: Standard, diverse VQA benchmarks covering compositional reasoning, knowledge-based reasoning, and general perception.
Key Insights and Ideas
- The bottleneck is visual perception, not reasoning: When zoomed in, BLIP-2 shifts from generic predictions ("white bird") to fine-grained identifications ("egret") without altering the text prompt.
- The decline is dramatic but reversible: In TextVQA, BLIP-2 (Flan-T5) accuracy drops by 46% between large targets ($S \ge 0.05$ of the image) and tiny targets ($S < 0.005$). Ground-truth cropping reduces this drop to under 19%.
- Architecture matters for visual token concatenation: Visual cropping reliably benefits encoder-decoder models (BLIP-2 Flan-T5 across all benchmarks). However, for decoder-only models (BLIP-2 OPT), concatenating crop tokens can degrade overall performance on general VQA datasets, likely because decoder-only self-attention expects visual tokens at specific prompt-initial positions.
- Task trade-offs: Cropping provides significant gains on attribute identification and text reading, but hurts questions that require holistic scene context, such as counting or spatial localization.
Key Results
- Fine-Detail Benchmarks:
- On FDVQA, base BLIP-2 Flan-T5 achieves 33.94% accuracy.
clip-CROPraises this to 36.61% (while ground-truthhuman-CROPachieves 42.29%). - On TextVQA, Flan-T5 baseline accuracy of 25.91% improves to 30.93% with
clip-CROPand 32.31% withsam-CROP(human-CROPreaches 37.68%). - General Benchmarks:
- Flan-T5 shows modest improvements across broad benchmarks: GQA increases from 43.85% to 45.23% (
sam-CROP), and A-OKVQA increases from 43.42% to 45.34% (clip-CROP). - Speed & Latency:
yolo-CROPis fastest (0.355s on an NVIDIA RTX A5000), butclip-CROP(1.072s) offers the best balance of speed and accuracy.sam-CROPis the heaviest (3.329s on GPU; >91s on CPU).
Limitations and Future Work
- The Human-Machine Gap: While automated methods help, a notable performance gap remains between automated heuristic cropping and human-annotated cropping.
- Global Context Loss: Merging crops via naive token concatenation harms tasks requiring holistic scene views (e.g., counting and spatial relations).
- Future Directions: Developing dynamic, question-conditioned routing (deciding when to crop) and more sophisticated cross-attention fusion mechanisms to better blend local crops with global views.
Summary updated