Bits, Not Boxes: Label-Free Layer Selection for Visual Localization in VLMs

Haisheng Fu, Gexin Huang, Haikai Zhao, Jie Liang, Zhenman Fang, Jingning Han, Xiaoxiao LiarXiv 2026 · arXiv

TL;DR

WhereBits is a label-free method for visual localization in Vision-Language Models (VLMs) that identifies which model layers are best at "looking" at objects without requiring any human-annotated bounding boxes. By using information theory to measure "where-to-look bits"—the redistribution of attention caused by a specific question—it matches the performance of supervised methods and significantly improves the accuracy of training-free image cropping.

What Problem Does This Paper Address?

Vision-Language Models (VLMs) like LLaVA or Qwen-VL use attention mechanisms to process images, but their raw attention maps are notoriously unreliable for pinpointing exactly where the model is "looking." A major culprit is the attention sink phenomenon: certain tokens (often the first visual token or corners) attract massive amounts of attention regardless of the prompt, drowning out the actual visual evidence.

While researchers have found that contrasting a specific question (e.g., "What color is the cat?") against a generic prompt (e.g., "Describe this image") can suppress these sinks, a new problem arises: layer selection. Not all layers in a deep VLM are equally good at localization. Some layers focus on low-level features, while others handle complex reasoning. Currently, finding the "good" layers requires expensive human-labeled bounding boxes for every new model or resolution. The authors of this paper ask: Can the model's own attention tell us which layers to trust, without any labels?

Attention sinks (crosses) pull standard attention maps away from the target (cyan box). WhereBits (rings) successfully localizes the evidence.

Methods

The core of WhereBits is a label-free estimator that weights attention heads and layers based on how much a question changes their "routing" behavior.

  1. Four Input Conditions: For a given image and question, the model performs forward passes under four conditions: (Image, Question), (Image, Generic Prompt), (Gray Image, Question), and (Gray Image, Generic Prompt).
  2. Measuring "Where-to-Look Bits": The authors use the KL divergence to measure how much a question redistributes a head's attention across visual tokens. They mathematically separate the effect of a question into two parts: whether to look at the image (total attention mass) and where to look (the distribution over tokens). Only the "where-to-look" part, measured in bits, is used for localization.
  3. Layer Prior: By averaging these "where-to-look bits" over a small set of unlabeled calibration questions (as few as 10), they create a layer prior. This prior identifies which layers consistently respond to questions by shifting their attention to specific image regions.
  4. Per-Head Contrastive Scores: Within each layer, they score individual tokens using the log-ratio of attention under the question vs. the generic prompt. This log-ratio is mathematically robust to attention sinks because shared additive terms in the attention logits cancel out.
The WhereBits pipeline: from raw attention rows to a question-specific localization map using information-theoretic weighting.

Datasets

The authors evaluated WhereBits across eight different VLMs from five model families (SmolVLM, InternVL3, Qwen3-VL, LLaVA-1.5, and Janus-Pro). They used:

  • COCO val2017: For both calibration (unlabeled) and evaluation of pointing accuracy.
  • RefCOCO: To test localization on referring expressions.
  • V* Bench: A benchmark for detecting small visual details.
  • TextVQA: To evaluate "zoom-in" performance on text-heavy images.

Key Insights and Ideas

  • Information Theory as a Filter: The "aha moment" is that localization signal isn't just about high attention, but about the change in attention distribution. Measuring this change in "bits" provides a principled way to weight different parts of the model.
  • Label-Free Calibration: You don't need to know where the object is to know if a layer is trying to find it. If a layer's attention distribution shifts significantly when you ask a specific question vs. a generic one, that layer is likely a "localizer."
  • Robustness to Sinks: The paper provides a formal proof (Proposition 1) showing that per-head log-ratios are invariant to shared logit offsets. This explains why WhereBits is so much better at ignoring the "sinks" that plague standard attention maps.
  • Generalization: The layer prior calculated on one set of unlabeled questions (e.g., from COCO) transfers effectively to entirely different tasks and even higher image resolutions.
Unlabeled "where-to-look" bits (blue) closely track the actual ground-truth localization accuracy (black) across model layers.

Key Results

  • Superior Localization: On COCO, WhereBits achieved a pointing accuracy of 0.72, compared to just 0.22 for standard mean attention.
  • Matches Supervised Baselines: WhereBits performs on par with (and often better than) methods like ViCrop that are explicitly tuned using human-labeled bounding boxes.
  • Effective Zoom-in: Using WhereBits to automatically crop and "zoom in" on relevant image regions raised V Bench accuracy by 6.0 points*.
  • Faithfulness: In "deletion-insertion" tests (where pixels are removed based on the importance map), WhereBits ranked first, meaning its maps more accurately reflect the visual features the model actually uses to answer questions.
WhereBits (blue) consistently outperforms other label-free and even some label-tuned methods in faithfulness rankings.

Limitations and Future Work

  • Small Targets: While better than previous methods, WhereBits still struggles with extremely small targets that might occupy only a single visual token.
  • Reference Prompt Sensitivity: The choice of the "generic prompt" matters. A prompt that is too similar to the question can reduce the contrastive signal.
  • Causal Decoders Only: The current mathematical proofs for sink-suppression rely on the layout of causal decoders (where images precede text). Extending this to other architectures might require adjustments.
  • Future Directions: The authors suggest that these "where-to-look bits" could be used during model training or for more efficient token pruning in high-resolution VLMs.

Summary updated

Abstract

Attention maps cheaply show where a vision-language model (VLM) looks, but attention sinks make them unreliable for localization. Contrasting the question with a generic prompt suppresses sinks, yet only some layers localize the queried object. Existing methods find these layers with box annotations or fixed rules. We show that attention itself can identify them. A question changes how much a head attends to the image and where this attention goes. The two effects separate exactly, and the second, measured in bits on unlabeled questions, indicates which layers localize. WhereBits turns these where-to-look bits into a layer prior and combines it with per-head contrastive scores. Its three design constants were fixed once on labeled COCO data, and a new model or resolution needs only unlabeled questions. Across eight VLMs, WhereBits matches the pointing accuracy of box-tuned layers on COCO and RefCOCO and ranks target tokens better. This advantage holds on a held-out model family and at higher resolutions. Its COCO pointing accuracy is 0.72, against 0.22 for mean attention. In training-free zoom-in, its crops raise V* accuracy by 6.0 points under forced-choice evaluation.
BibTeX
@misc{fu2026bits,
  author = {Haisheng Fu and Gexin Huang and Haikai Zhao and Jie Liang and Zhenman Fang and Jingning Han and Xiaoxiao Li},
  title = {{Bits, Not Boxes: Label-Free Layer Selection for Visual Localization in VLMs}},
  year = {2026},
  eprint = {2611.23432},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23432},
}
Figure