High-Norm Outlier Tokens in Vision Transformers

34 concepts23 papers1 proposal

When deep Vision Transformers (ViTs) are trained on large datasets, a peculiar numerical anomaly consistently emerges in their deeper layers: a tiny fraction of patch tokens suddenly develop disproportionately large feature norms. These extreme values do not appear uniformly across the image. Instead, they consistently manifest in low-information background regions, such as flat skies or empty walls. Rather than representing critical local features, these high-norm tokens act as "attention sinks" or computational artifacts that dominate the network's internal representations.

The Scratchpad Hypothesis: Why Outliers Form

Why does a model allocate its highest numerical energy to the least informative parts of an image? As Timothée Darcet et al. demonstrated in Vision Transformers Need Registers, the standard transformer architecture lacks an explicit internal memory or "scratchpad" to discard redundant local information and store global scene statistics. Because background patches contain little unique semantic content, the model "hijacks" them to aggregate global context.

Mathematically, this aggregation behaves like a power iteration driven by the network's own weights. As shown by Haoqi Wang et al. in SINDER: Repairing the Singular Defects of DINOv2, these "singular defects" cause specific token activations to explode across deep layers. This amplification is further exacerbated by standard normalization layers; Ruihao Gong et al. noted in Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models that the scaling parameters in LayerNorm act as amplifiers for these extreme values.

This phenomenon is remarkably pervasive, extending beyond standard ViTs to alternative architectures:

How Outliers Disrupt Downstream Tasks

While these outlier tokens allow the model to track global context, their extreme magnitudes create severe side effects due to the nature of the softmax attention mechanism. A token with an anomalously high norm draws an excessive share of attention, effectively drowning out the subtle local signals required for precise spatial understanding.

This pollution of the feature space degrades performance across several downstream tasks:

Architectural and Test-Time Solutions

To resolve this issue, researchers have developed two primary paradigms: architectural modification via dedicated registers, and test-time suppression.

Dedicated Register Tokens

The most elegant architectural solution is to explicitly provide the model with the scratchpad memory it seeks. By appending dedicated, non-spatial "register tokens" to the input sequence, the model learns to store global information in these designated slots rather than hijacking background patches. Timothée Darcet et al. demonstrated that this simple addition completely eliminates high-norm artifacts from the spatial feature maps.

Interestingly, these registers do not just act as trash cans; Kowshik Thopalli et al. showed in Leveraging Registers in Vision Transformers for Robust Adaptation that the resulting register embeddings contain rich global information that significantly improves out-of-distribution robustness. This register strategy has also been successfully adapted to state-space architectures by Feng Wang et al. in Mamba-Reg: Vision Mamba Also Needs Registers, enabling the first stable scaling of Vision Mamba models to large parameter regimes.

Test-Time Suppression and Calibration

For models that are already pre-trained, retraining with registers from scratch is computationally prohibitive. To address this, Nicholas Jiang et al. introduced a training-free method in Vision Transformers Don't Need Trained Registers that isolates the sparse set of "register neurons" responsible for the outliers and dynamically shifts their activations into a new test-time register token.

Alternatively, inference-time calibration can bypass the artifacts entirely. Donguk Kim in Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models proposed a plug-and-play decoding strategy that dynamically detects these "blind tokens" by identifying attention scores that deviate significantly from the mean, subsequently recalibrating the model's focus to mitigate hallucinations.

Key Foundational Work

Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models (2022) — contributed 1 concept elements to this direction.
Transferable Adversarial Attacks on Vision Transformers with Token Gradient Regularization (2023) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_high_norm_outlier_tokens_96c5b35e,
  title = {high-norm outlier tokens},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-norm-outlier-tokens/rdr_96c5b35e4970413c85c20af8fb6546a9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_high_norm_outlier_tokens_96c5b35e_v1,
  title = {high-norm outlier tokens},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-norm-outlier-tokens/rdr_96c5b35e4970413c85c20af8fb6546a9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake