Low-Pass Filtering Bias in Self-Attention Mechanisms

125 concepts79 papers

When we think about why Transformers excel at complex tasks, we often point to their ability to capture long-range dependencies or global context. However, looking at the architecture through the lens of signal processing reveals a fundamental, structural bias: the self-attention mechanism inherently acts as a low-pass filter.

To understand why, we have to look at the core mathematical operation of attention. For any given layer, the model computes an attention matrix by applying a softmax normalization to the scaled dot-product of queries and keys:

$$A = \text{softmax}\left(\frac{QK^T}{\sqrt{d}}\right)$$

This attention matrix $A$ is then multiplied by the value matrix $V$. Crucially, because of the softmax operation, each row in $A$ sums to 1. This means that computing $AV$ is fundamentally an operation of weighted averaging across tokens.

In spatial or sequential data, taking a weighted average of neighboring representations is a smoothing operator. In the frequency domain, smoothing is the definition of a low-pass filter: it preserves slow-varying, global structures (low frequencies) while attenuating sharp, rapid variations (high frequencies). As shown by the authors of How Do Vision Transformers Work?, multi-head self-attention mechanisms function primarily as generalized spatial smoothing operators. This stands in sharp contrast to traditional Convolutional Neural Networks (CNNs), which naturally tend to amplify high-frequency signals to capture local textures and sharp edges.

Dmitrii Marin formalized this behavior in Token Pooling in Vision Transformers, modeling self-attention as a high-dimensional low-pass filter akin to Gaussian smoothing. While this smoothing is excellent for capturing broad context, it introduces a systematic vulnerability when stacked across deep architectures.

The Cascading Cost of Depth and Length

What happens when a network passes data through 12, 24, or 32 layers of consecutive low-pass filters? The high-frequency signals do not simply get dampened; they are progressively erased. This phenomenon manifests across different modalities as a loss of fine-grained information.

1. Feature Collapse and Blurry Boundaries in Vision

In deep Vision Transformers (ViTs), this continuous smoothing leads to "over-smoothing" or "feature collapse," where token representations become nearly identical. Peihao Wang et al. provided a rigorous Fourier-domain proof of this in Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice, demonstrating that softmax normalization forces feature maps to converge toward a uniform global average.

This spectral decay makes it incredibly difficult for deep models to preserve sharp visual details:

2. Embedding and Sequence Collapse in Language and Recommendations

The low-pass filtering effect is not unique to vision; it scales aggressively with sequence length. In Length-Induced Embedding Collapse in PLM-based Models, Sunhao Dai et al. revealed that as input sequences grow longer, the self-attention mechanism smooths out unique token-level details to preserve a global average signal, causing long-text embeddings to lose their distinctiveness and collapse into a narrow space.

Similarly, in sequential recommendation systems, Xinyu Du et al. noted in Frequency Enhanced Hybrid Attention Network for Sequential Recommendation that self-attention prioritizes global trends at the expense of high-frequency signals, effectively washing away rapid, short-term shifts in user behavior.

The Silver Lining: Noise Suppression

Interestingly, this low-pass bias is not always a flaw. Hidenori Tanaka et al. proved in Provable Low-Frequency Bias of In-Context Learning of Representations that this exact low-pass filtering property is what allows Transformers to perform in-context learning effectively. By acting as a low-pass filter on input sequences, the model suppresses high-frequency noise and forces hidden representations to converge toward the smooth, global geometry of the underlying data distribution.

Counteracting the Bias: Spectral Interventions

To build models that retain both global context and sharp local details, researchers have developed several clever interventions to counteract or bypass the low-pass attention bias. These can be categorized into three main strategies.

1. Hybrid Architectures and Frequency-Aware Mixers

Instead of forcing self-attention to capture high frequencies, hybrid models split the workload. Chenyang Si et al. designed the Inception Transformer, which splits channels into parallel paths: a convolutional path to capture high-frequency local details (edges and textures) and a self-attention path to capture low-frequency global structures.

Other approaches replace or augment attention with wavelet transforms to prevent information loss during downsampling. Yuetong FANG et al. introduced the Spiking Wavelet Transformer, swapping out global self-attention for a Haar wavelet-based token mixer to explicitly preserve sharp visual transitions. Similarly, the authors of Wavelet-based Bi-dimensional Aggregation Network for SAR Image Change Detection replaced traditional downsampling pooling layers with Discrete Wavelet Transforms (DWT) to prevent the permanent, "leaky" loss of high-frequency details during feature matching.

2. Attention Scaling and Residual Recycling

If the softmax operation is mathematically responsible for over-smoothing, we can modify the attention computation to preserve high frequencies. Peihao Wang et al. introduced AttnScale and FeatScale in Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice to dynamically amplify high-frequency signals as the network deepens. To combat length-induced collapse, Sunhao Dai et al. developed TempScale in Length-Induced Embedding Collapse in PLM-based Models, which uses a temperature-scaling factor to sharpen the attention distribution for long sequences, keeping token representations distinct.

Alternatively, rather than preventing the smoothing, models can capture what the filter discards. Laziz Abdullaev et al. introduced Twicing Attention in Transformer Meets Twicing: Harnessing Unattended Residual Information. This approach calculates the residual information left behind by the attention smoothing operator and re-injects it back into the network, effectively recycling the high-frequency details that would otherwise be lost.

3. Frequency-Domain Supervision and Guidance

A final strategy involves forcing the model to value high frequencies through targeted training objectives or explicit prompts. Jiawang Bai et al. proposed High-frequency Adversarial Training (HAT) in Improving Vision Transformers by Revisiting High-Frequency Components, using adversarial perturbations targeted at edges and textures to force the ViT to prioritize high-frequency components during optimization.

Standard Attention:
Input Signal ───► [ Softmax Smoothing (Low-Pass Filter) ] ───► Blurred/Global Features

Frequency Prompting (FPro):
               ┌─► Low-Frequency Component  ──► Standard Attention ──┐
Input Signal ──┤                                                     ├──► Restored Image
               └─► High-Frequency Component ──► Frequency Prompt ────┘

For image restoration tasks, Jinshan Pan introduced Seeing the Unseen: A Frequency Prompt Guided Transformer for Image Restoration. This method explicitly decouples the input image into high- and low-frequency channels, using the high-frequency components as adaptive "prompts" to guide the transformer layers, ensuring that fine textures and sharp edges are accurately reconstructed despite the attention mechanism's natural inclination to smooth them away.

Key Foundational Work

Improving Vision Transformers by Revisiting High-Frequency Components (2022) — contributed 3 concept elements to this direction.
Inception Transformer (2022) — contributed 2 concept elements to this direction.
Inception Transformer (2022) — contributed 2 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 81 papers
  2. Version 1 · 2026-10-04 · 140 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_high_frequency_information_loss_ff13aa98,
  title = {high-frequency information loss},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-frequency-information-loss/rdr_ff13aa98fbd24054929192a8fbb01049},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_high_frequency_information_loss_ff13aa98_v2,
  title = {high-frequency information loss},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-frequency-information-loss/rdr_ff13aa98fbd24054929192a8fbb01049/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}

See something wrong? Report a mistake