When deep Vision Transformers (ViTs) are trained on large datasets, a peculiar numerical anomaly consistently emerges in their deeper layers: a tiny fraction of patch tokens suddenly develop disproportionately large feature norms. These extreme values do not appear uniformly across the image. Instead, they consistently manifest in low-information background regions, such as flat skies or empty walls. Rather than representing critical local features, these high-norm tokens act as "attention sinks" or computational artifacts that dominate the network's internal representations.
The Scratchpad Hypothesis: Why Outliers Form
Why does a model allocate its highest numerical energy to the least informative parts of an image? As Timothée Darcet et al. demonstrated in Vision Transformers Need Registers, the standard transformer architecture lacks an explicit internal memory or "scratchpad" to discard redundant local information and store global scene statistics. Because background patches contain little unique semantic content, the model "hijacks" them to aggregate global context.
Mathematically, this aggregation behaves like a power iteration driven by the network's own weights. As shown by Haoqi Wang et al. in SINDER: Repairing the Singular Defects of DINOv2, these "singular defects" cause specific token activations to explode across deep layers. This amplification is further exacerbated by standard normalization layers; Ruihao Gong et al. noted in Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models that the scaling parameters in LayerNorm act as amplifiers for these extreme values.
This phenomenon is remarkably pervasive, extending beyond standard ViTs to alternative architectures:
- State-Space Models: In Mamba-R: Vision Mamba ALSO Needs Registers, Feng Wang et al. observed that background artifacts act as a primary bottleneck preventing recurrent vision models from scaling effectively to large parameter regimes.
- Diffusion Transformers: In Unleashing Diffusion Transformers for Visual Correspondence by Modulating Massive Activations, Chaofan Gan et al. identified "massive activations" dominating specific feature dimensions across all tokens, which severely degrades the model's spatial understanding.
How Outliers Disrupt Downstream Tasks
While these outlier tokens allow the model to track global context, their extreme magnitudes create severe side effects due to the nature of the softmax attention mechanism. A token with an anomalously high norm draws an excessive share of attention, effectively drowning out the subtle local signals required for precise spatial understanding.
This pollution of the feature space degrades performance across several downstream tasks:
- Dense Spatial Prediction: In tasks like semantic segmentation, depth estimation, and object discovery, smooth and localized attention maps are essential. The authors of Vision Transformers with Self-Distilled Registers (Zipeng Yan et al.) and Improving Visual Discriminability of CLIP for Training-Free Open-Vocabulary Semantic Segmentation (Jinxin Zhou et al.) found that these artifacts mask local semantics and distort spatial coherence. Similarly, Rémi Kazmierczak et al. showed in Enhancing Concept Localization in CLIP-based Concept Bottleneck Models that aggregating global statistics into local patch tokens severely undermines concept localization.
- Multimodal Alignment and Explainability: When visual encoders are integrated into larger systems, these artifacts introduce significant noise. For example, Edson Araujo et al. noted in CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment that non-semantic sinks obscure fine-grained audio-visual mapping. Furthermore, in explainability frameworks, Walid Bousselham et al. demonstrated in DEX-AR: A Dynamic Explainability Method for Autoregressive Vision-Language Models that these artifacts exhibit deceptively high attention values, distorting attention-based explanations.
- Model Hallucinations: In Large Vision-Language Models (LVLMs), these background outliers trigger a vulnerability known as "attention hijacking." As detailed by Xuan Gong et al. in DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination and Xinyu Lyu et al. in Attention Hijackers: Detect and Disentangle Attention Hijacking in LVLMs for Hallucination Mitigation, the language decoder can fixate on these high-norm background distractors rather than discriminative object details, leading the model to hallucinate objects that are not present.
- Knowledge Distillation and Tuning: The presence of these outliers also corrupts the learning signal during model training. Geunhyeok Yu et al. argued in SiNGER: A Clearer Voice Distills Vision Transformers Further that these artifacts distract student models during knowledge distillation, while Chen Chen et al. in DA-VPT: Semantic-Guided Visual Prompt Tuning for Vision Transformers showed they degrade prompt tuning by passing uninformative background noise to the classification token.
Architectural and Test-Time Solutions
To resolve this issue, researchers have developed two primary paradigms: architectural modification via dedicated registers, and test-time suppression.
Dedicated Register Tokens
The most elegant architectural solution is to explicitly provide the model with the scratchpad memory it seeks. By appending dedicated, non-spatial "register tokens" to the input sequence, the model learns to store global information in these designated slots rather than hijacking background patches. Timothée Darcet et al. demonstrated that this simple addition completely eliminates high-norm artifacts from the spatial feature maps.
Interestingly, these registers do not just act as trash cans; Kowshik Thopalli et al. showed in Leveraging Registers in Vision Transformers for Robust Adaptation that the resulting register embeddings contain rich global information that significantly improves out-of-distribution robustness. This register strategy has also been successfully adapted to state-space architectures by Feng Wang et al. in Mamba-Reg: Vision Mamba Also Needs Registers, enabling the first stable scaling of Vision Mamba models to large parameter regimes.
Test-Time Suppression and Calibration
For models that are already pre-trained, retraining with registers from scratch is computationally prohibitive. To address this, Nicholas Jiang et al. introduced a training-free method in Vision Transformers Don't Need Trained Registers that isolates the sparse set of "register neurons" responsible for the outliers and dynamically shifts their activations into a new test-time register token.
Alternatively, inference-time calibration can bypass the artifacts entirely. Donguk Kim in Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models proposed a plug-and-play decoding strategy that dynamically detects these "blind tokens" by identifying attention scores that deviate significantly from the mean, subsequently recalibrating the model's focus to mitigate hallucinations.
Key Foundational Work
Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models (2022) — contributed 1 concept elements to this direction.
Transferable Adversarial Attacks on Vision Transformers with Token Gradient Regularization (2023) — contributed 1 concept elements to this direction.