TL;DR
Multimodal Large Language Models (MLLMs) suffer from a significant "perception gap" when dealing with small visual details, with zero-shot accuracy dropping by up to 46% as object size decreases. This paper demonstrates that this failure is causal—simply "zooming in" via cropping fixes it—and introduces five automated cropping methods that boost performance by leveraging either external models or the MLLM’s own internal attention and gradients.
What Problem Does This Paper Address?
While MLLMs like BLIP-2 and GPT-4 have shown impressive zero-shot capabilities, they often stumble on questions requiring fine-grained visual perception, such as reading a small street sign or identifying a tiny bird species.
The authors set out to answer two questions:
- How sensitive are these models to the physical size of the subject in a question?
- Is the failure due to a lack of reasoning/knowledge or a fundamental limitation in visual perception?
By partitioning datasets based on object size, they found a massive performance cliff. However, they also discovered that if a human manually crops the image to the relevant area, the model’s accuracy skyrockets. This suggests the models have the "intelligence" to answer the question but lack the "eyesight" to resolve small details at standard input resolutions.
Methods
The researchers propose five automatic visual cropping methods to find the "Region of Interest" (RoI) at inference time:
External Knowledge Methods
- clip-CROP: Iteratively crops the image from four sides, using CLIP to select the crop with the highest semantic similarity to the question.
- yolo-CROP: Uses a YOLOv8 object detector to find candidate objects and selects the one most relevant to the question via CLIP.
- sam-CROP: Uses the Segment Anything Model (SAM) to generate granular masks, converting them to bounding boxes for selection.
Native Visual Cropping (The "Aha!" Approach)
These methods are particularly novel because they don't require extra models; they look at the MLLM’s own "thought process":
- grad-CROP: Tracks the gradient of the model’s decision back to the input pixels to see which area "triggered" the answer.
- att-CROP: Aggregates attention scores from the model’s Transformer layers (specifically the Q-Former and LLM) to identify where the model is internally focusing.
Key Insights and Ideas
- The Perception Bottleneck is Causal: The fact that cropping (without changing the question or the model) restores accuracy proves that the limitation is in the visual front-end, not the linguistic reasoning.
- Models Know "Where," but not "What": The success of
att-CROPshows that even when a model gives the wrong answer, its internal attention is often already focused on the correct region. It just can't "see" clearly enough to identify the object until it is zoomed in. - The Global-Local Trade-off: Cropping is not a silver bullet. While it helps with "What color is the bird?", it can hurt "How many birds are there?" because the model loses the global context. The authors suggest a "question-gate" to decide when to crop based on the question type.
Key Results
- Massive Accuracy Gains: On the TextVQA dataset, human-level cropping improved BLIP-2 FlanT5 accuracy from 25.91% to 37.68%.
- Native Methods Compete: The
att-CROPmethod (using the model's own attention) achieved 34.26% on TextVQA, significantly outperforming the baseline without needing an external detector like YOLO. - FDVQA Success: On the authors' new fine-detail dataset (FDVQA),
clip-CROPimproved accuracy from 33.94% to 36.61%, while human cropping reached 42.29%.
Limitations and Future Work
- The "Why" Remains Elusive: While the paper proves the perception limit exists, it doesn't fully explain why architectures with high-resolution encoders (like ViT) still fail to resolve these details in a zero-shot setting.
- Fusing Scales: The current approach simply concatenates original and cropped tokens. The authors note that more sophisticated multi-scale fusion mechanisms are needed to maintain global context while perceiving local details.
Summary updated