TL;DR
WhereBits is a label-free method for visual localization in Vision-Language Models (VLMs) that identifies which model layers are best at "looking" at objects without requiring any human-annotated bounding boxes. By using information theory to measure "where-to-look bits"—the redistribution of attention caused by a specific question—it matches the performance of supervised methods and significantly improves the accuracy of training-free image cropping.
What Problem Does This Paper Address?
Vision-Language Models (VLMs) like LLaVA or Qwen-VL use attention mechanisms to process images, but their raw attention maps are notoriously unreliable for pinpointing exactly where the model is "looking." A major culprit is the attention sink phenomenon: certain tokens (often the first visual token or corners) attract massive amounts of attention regardless of the prompt, drowning out the actual visual evidence.
While researchers have found that contrasting a specific question (e.g., "What color is the cat?") against a generic prompt (e.g., "Describe this image") can suppress these sinks, a new problem arises: layer selection. Not all layers in a deep VLM are equally good at localization. Some layers focus on low-level features, while others handle complex reasoning. Currently, finding the "good" layers requires expensive human-labeled bounding boxes for every new model or resolution. The authors of this paper ask: Can the model's own attention tell us which layers to trust, without any labels?
Methods
The core of WhereBits is a label-free estimator that weights attention heads and layers based on how much a question changes their "routing" behavior.
- Four Input Conditions: For a given image and question, the model performs forward passes under four conditions: (Image, Question), (Image, Generic Prompt), (Gray Image, Question), and (Gray Image, Generic Prompt).
- Measuring "Where-to-Look Bits": The authors use the KL divergence to measure how much a question redistributes a head's attention across visual tokens. They mathematically separate the effect of a question into two parts: whether to look at the image (total attention mass) and where to look (the distribution over tokens). Only the "where-to-look" part, measured in bits, is used for localization.
- Layer Prior: By averaging these "where-to-look bits" over a small set of unlabeled calibration questions (as few as 10), they create a layer prior. This prior identifies which layers consistently respond to questions by shifting their attention to specific image regions.
- Per-Head Contrastive Scores: Within each layer, they score individual tokens using the log-ratio of attention under the question vs. the generic prompt. This log-ratio is mathematically robust to attention sinks because shared additive terms in the attention logits cancel out.
Datasets
The authors evaluated WhereBits across eight different VLMs from five model families (SmolVLM, InternVL3, Qwen3-VL, LLaVA-1.5, and Janus-Pro). They used:
- COCO val2017: For both calibration (unlabeled) and evaluation of pointing accuracy.
- RefCOCO: To test localization on referring expressions.
- V* Bench: A benchmark for detecting small visual details.
- TextVQA: To evaluate "zoom-in" performance on text-heavy images.
Key Insights and Ideas
- Information Theory as a Filter: The "aha moment" is that localization signal isn't just about high attention, but about the change in attention distribution. Measuring this change in "bits" provides a principled way to weight different parts of the model.
- Label-Free Calibration: You don't need to know where the object is to know if a layer is trying to find it. If a layer's attention distribution shifts significantly when you ask a specific question vs. a generic one, that layer is likely a "localizer."
- Robustness to Sinks: The paper provides a formal proof (Proposition 1) showing that per-head log-ratios are invariant to shared logit offsets. This explains why WhereBits is so much better at ignoring the "sinks" that plague standard attention maps.
- Generalization: The layer prior calculated on one set of unlabeled questions (e.g., from COCO) transfers effectively to entirely different tasks and even higher image resolutions.
Key Results
- Superior Localization: On COCO, WhereBits achieved a pointing accuracy of 0.72, compared to just 0.22 for standard mean attention.
- Matches Supervised Baselines: WhereBits performs on par with (and often better than) methods like ViCrop that are explicitly tuned using human-labeled bounding boxes.
- Effective Zoom-in: Using WhereBits to automatically crop and "zoom in" on relevant image regions raised V Bench accuracy by 6.0 points*.
- Faithfulness: In "deletion-insertion" tests (where pixels are removed based on the importance map), WhereBits ranked first, meaning its maps more accurately reflect the visual features the model actually uses to answer questions.
Limitations and Future Work
- Small Targets: While better than previous methods, WhereBits still struggles with extremely small targets that might occupy only a single visual token.
- Reference Prompt Sensitivity: The choice of the "generic prompt" matters. A prompt that is too similar to the question can reduce the contrastive signal.
- Causal Decoders Only: The current mathematical proofs for sink-suppression rely on the layout of causal decoders (where images precede text). Extending this to other architectures might require adjustments.
- Future Directions: The authors suggest that these "where-to-look bits" could be used during model training or for more efficient token pruning in high-resolution VLMs.
Summary updated