Training Vision Transformers to Focus: Emergence of Spatial Attention Head Specialization

Yuji Nozawa, Yu-Chieh Lin, Youyang NgarXiv 2026 · arXiv

TL;DR

Researchers from Kioxia Corporation have introduced two novel "visual focusing" loss functions that force self-supervised Vision Transformers (ViTs) to develop spatially specialized attention heads. By establishing a direct correspondence between specific image regions and model attention during training, they significantly improve the interpretability and performance of models like DINOv2 on dense prediction tasks like zero-shot image segmentation.

What Problem Does This Paper Address?

Self-supervised learning (SSL) has allowed models like DINO and DINOv2 to learn incredibly rich visual representations without a single human label. However, these models have a "redundancy" problem: because they lack an explicit mechanism to guide their focus during training, many of their multi-head attention units end up learning very similar patterns.

In practice, this often manifests as a "center bias," where most attention heads focus on the dominant object in the middle of the frame. While great for global image classification, this lack of diversity makes the models less effective for "dense" tasks—like identifying multiple distinct objects or segmenting complex scenes—and limits our ability to interpret what the model is actually "seeing" in different parts of an image.

Methods

The authors propose a framework called Region-to-Focus Correspondence (ReFoCo). They build upon the DINOv2 architecture (specifically the version with "registers" to reduce artifacts) and introduce two new loss functions:

  1. Inside-focus loss ($L_{in}$): This loss encourages the student network to align its attention on a specific "Region of Interest" (ROI) with a teacher network that is only looking at a crop of that same ROI.
  2. Outside-focus loss ($L_{out}$): This forces the model to focus on regions outside a masked box. It encourages the model to find informative features in the periphery, preventing it from solely relying on the most discriminative central parts of an object.

To implement this, they use a technique called Prompt-guided Attention Head Selection (PHS). During training, the model selects the heads that naturally react most strongly to a random ROI and then "steers" the training signal to further specialize those heads for that spatial area.

Figure 2: Overview of our proposed visual focusing loss functions Lin and Lout.

Datasets

The models were trained from scratch on the ImageNet-1k dataset (without labels). Evaluation was performed across several benchmarks to test different capabilities:

  • Zero-shot Segmentation: ImageNet-Segmentation and PASCAL VOC.
  • Unsupervised Object Discovery: VOC07, VOC12, and COCO 20k.
  • Image Retrieval: COCO and PASCAL VOC.
  • Classification: ImageNet-1k (to check for performance trade-offs).

Key Insights and Ideas

  • Emergent Specialization: The most striking finding is that spatial specialization emerges without manual annotations. By simply providing random boxes during training, the model learns to assign different heads to different spatial "slots."
  • Complementary Roles: $L_{in}$ and $L_{out}$ are not redundant. $L_{in}$ helps the model focus on peripheral objects that the baseline might ignore, while $L_{out}$ helps the model "see around" central objects to capture the broader context.
  • Interpretability via Diversity: In a standard DINOv2 model, if you show it a box of four donuts, almost all heads look at the center of the box. In the ReFoCo-trained model, individual heads specialize: Head 1 might look at the top-left donut, while Head 7 focuses on the bottom-right one.
Figure 1: Comparison of attention maps produced by the DINOv2 trained model and our trained model. For each model, we show the attention maps averaged over the validation set of the ImageNet- 1k (Russakovsky et al., 2015) and the attention maps obtained from a single example image. Existing learned perceptual representations (left) in individual attention heads often tend to be ambiguous or overly similar within the example image and across images in the dataset, which limits their interpretability. In contrast, our method (right) selectively steers the focus, leading to the emergence of diverse and spatially focused perceptual capabilities. (Images: Visualization of 16 attention heads in the last layer of trained ViT-L/16-Reg model)

Key Results

  • Segmentation Boost: On ImageNet-Segmentation, the proposed model achieved a 67.29% mIoU, a significant jump from the baseline DINOv2's 63.67%.
  • Object Discovery: The model outperformed DINOv2 in unsupervised object discovery (LOST) across all tested datasets (VOC07, VOC12, and COCO).
  • The "Donut" Test: Qualitative analysis (Figure 5) shows that while the baseline model produces ambiguous, overlapping attention maps for a multi-object image, the ReFoCo model produces clean, distinct maps for individual objects.
Figure 5: An example of attention maps for each model (a)-(d). The image in the top left is the “input”. The images labeled “mean” represent the head means of the attention maps used as heat maps, and the images labeled “mask” represent the segmentation masks.

Limitations and Future Work

  • Classification Trade-off: The authors noted a slight drop (approx. 1–3%) in top-1 accuracy on ImageNet classification. This suggests that spatial specialization comes at a small cost to global feature aggregation.
  • Implicit Control: While the heads become specialists, the user cannot choose which head focuses where during training; it is determined implicitly.
  • Training Overhead: The specialized training process takes longer than standard DINOv2 (e.g., ~5.5 days vs ~3 days on 4x H200 GPUs).
  • Future Directions: The authors suggest exploring how these specialized heads could be dynamically enabled or disabled during inference for "on-the-fly" model customization in robotics or autonomous driving.

Summary updated

Abstract

Self-supervised learning in Vision Transformer models, such as DINO, enables robust visual perception without requiring labeled data by leveraging large-scale image datasets. However, these models lack a focusing mechanism during training, resulting in perceptual representations that are often ambiguous or overly similar. In this work, we introduce two visual focusing loss functions designed to establish correspondence between the model's attention and specific regions within images. Specifically, we leverage the multi-head attention mechanism within the model to selectively steer the model's focus, leading to the emergence of spatial attention head specialization, characterized by diverse and spatially focused perceptual capabilities without requiring annotations. Through both qualitative and quantitative evaluations, we demonstrate that our method substantially increases the spatial selectivity and diversity of attention heads, improving both interpretability and performance on dense prediction tasks such as zero-shot image segmentation.
BibTeX
@misc{nozawa2026training,
  author = {Yuji Nozawa and Yu-Chieh Lin and Youyang Ng},
  title = {{Training Vision Transformers to Focus: Emergence of Spatial Attention Head Specialization}},
  year = {2026},
  eprint = {2611.11890},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11890},
}
Figure