When visualizing the attention maps of large-scale Vision Transformers (ViTs), researchers often find something unsettling: a handful of "spiky" tokens in the background that have massive numerical norms. These tokens don't represent objects or textures; instead, they appear in uninformative regions like the sky or a blurry corner.
While these artifacts don't seem to hurt global image classification, they wreak havoc on dense prediction tasks like semantic segmentation and depth estimation. Recent research has revealed that these outliers are not random noise, but a fundamental architectural "hijacking" where the model repurposes background patches as internal scratchpads for global information.
The Hijacking Phenomenon
In a standard Transformer, every token must attend to every other token. As models grow in capacity and training duration, they develop a need to store and manipulate global context—information that describes the whole image rather than a specific patch.
Without a dedicated space for this, Timothée Darcet, Maxime Oquab, and their colleagues at Meta FAIR discovered that ViTs "hijack" low-information background patches to serve as internal storage. In Vision Transformers Need Registers, they explain that these patches become "registers" where the model aggregates global statistics. Because these patches contain little unique visual information, the model can overwrite their original features with global data without losing much local detail.
However, this comes at a cost. These hijacked tokens develop disproportionately high norms, acting as "attention sinks" that dominate the feature map. In tasks like unsupervised object discovery, these sinks create noisy, fragmented attention maps that fail to delineate object boundaries.
Why Sinks Form
The emergence of these artifacts appears to be a byproduct of the network's own weights. Haoqi Wang and Mathieu Salzmann argue in SINDER: Repairing the Singular Defects of DINOv2 that these are "singular defects" caused by the weights acting like a mathematical power iteration, amplifying specific feature dimensions until they become outliers.
This isn't limited to standard ViTs. Chaofan Gan and Xi Chen found similar "massive activations" in Diffusion Transformers (DiTs) like SD3 and Flux. In Unleashing Diffusion Transformers for Visual Correspondence by Modulating Massive Activations, they show that these outliers dominate feature vectors so heavily that standard similarity metrics fail, making it impossible for the model to match related points across different images.
Impact on Multimodal Models
In Large Vision-Language Models (LVLMs), these attention sinks contribute to "hallucinations." When a visual encoder like CLIP produces high-norm outlier tokens, they can "hijack" the language decoder's attention.
Xinyu Lyu identifies this as "Attention Hijacking," where instruction tokens (like "Please") inadvertently focus the model on these irrelevant background sinks rather than the actual objects in the image. Similarly, Sangmin Woo and Donguk Kim describe these as "blind tokens" in Don't Miss the Forest for the Trees, noting that they hog attention resources and distract the model from task-relevant features.
Solutions: From Explicit Registers to Post-Hoc Repair
The most direct solution is to give the model what it clearly wants: dedicated storage. By adding a few non-learnable "register tokens" to the input sequence—similar to the [CLS] token but without a specific task—Darcet et al. found that the model naturally uses these for global context, leaving the image patches clean and semantically meaningful.
For models that are already trained, researchers have developed several "repair" strategies:
- Training-Free Redistribution: Methods like VAR (Seil Kang et al.) and AVISC (Sangmin Woo et al.) identify these sinks at test-time and mathematically redistribute their attention weights back to informative image regions.
- Post-Hoc Distillation: Geunhyeok Yu and Sunjae Jeong proposed SiNGER (SiNGER: A Clearer Voice Distills Vision Transformers Further), which uses a lightweight adapter to "clean" a teacher model's features before passing them to a student, ensuring the student doesn't learn to replicate the teacher's artifacts.
- Self-Distilled Registers: In Vision Transformers with Self-Distilled Registers, Zipeng Yan and colleagues introduce PH-Reg, which adds register tokens to pre-trained ViTs through a short fine-tuning process, effectively "migrating" the artifacts from the image patches to the new registers.
By addressing these artifacts, researchers are moving toward Vision Transformers that are not only more accurate but also more interpretable, with attention maps that finally align with the physical structures of the visual world.