When we train deep neural networks to process images, audio, or time-series data, we typically force them to operate in the spatial or temporal domain. In these domains, a model perceives data through local windows—like a convolutional kernel sliding across pixels or a recurrent cell stepping through time. To capture long-range or global dependencies, networks must stack dozens of layers or employ self-attention mechanisms. However, self-attention incurs a quadratic computational cost ($O(N^2)$) and is notoriously sensitive to high-frequency noise.
An alternative approach shifts the perspective entirely by transforming data into the frequency domain using tools like the Fast Fourier Transform (FFT) or Discrete Cosine Transform (DCT). In the frequency domain, global patterns become compact spectral coordinates, and complex global convolutions simplify into efficient element-wise multiplications ($O(N \log N)$).
Rather than relying on traditional, handcrafted mathematical filters (such as a fixed low-pass or band-pass filter), a growing body of research focuses on learnable frequency domain filtering. By treating the frequency spectrum as a differentiable space, models can dynamically learn exactly which frequencies to amplify, attenuate, or mask to optimize for a specific task.
Parameterizing the Spectrum
To build a learnable frequency filter, a model must first map its input or hidden representations into the spectral domain, apply a parameterized transformation, and invert the signal back to the original domain. The core design challenge lies in how to parameterize this transformation so that it can be optimized via backpropagation.
Input Signal ──> [ FFT / DCT ] ──> Spectral Representation
│
▼
[ Learnable Filter ] <── Optimized via SGD
│
▼
Output Signal <── [ iFFT / iDCT ] <── Filtered Spectrum
The simplest approach is to multiply the frequency spectrum by a learnable tensor of weights. For example, Yuxing Tian demonstrates that multiplying frequency-domain signals by a learnable complex number tensor allows a model to adaptively tune into significant periodic components while filtering out noise.
Other architectures parameterize the boundaries or shapes of the filters themselves:
- Learnable Cutoffs: Zhaoyang Cong et al. design adaptive bandpass filters where the cutoff frequencies are treated as learnable parameters. This allows the network to dynamically isolate task-relevant physiological frequencies during training.
- Differentiable Gating: Szu-Wei Fu et al. introduce a per-frequency learnable sigmoid function with a scaling factor above unity, providing the flexibility required to handle complex phase differences during signal reconstruction. Similarly, Youngin Park et al. utilize sigmoid-activated weights and thresholding to adaptively prune or preserve specific frequency components for anti-aliasing.
- Trainable Linear Transformations: Instead of fixed basis functions, Zhong-Qiu Wang et al. implement frequency decomposition as a differentiable linear transformation layer, enabling the system to learn an optimal filterbank configuration tailored to specific environmental noise.
Dynamic and Content-Adaptive Filtering
While static learnable filters apply the same spectral mask to every sample in a dataset, dynamic frequency filtering allows the filter coefficients to change based on the specific input instance. This turns the network into a content-adaptive equalizer.
Zhipeng Huang et al. show that dynamic frequency filters that adapt to specific input instances significantly outperform static filters. By adjusting the spectral focus based on image content, these models achieve the representational power of large, image-sized spatial convolution kernels at a fraction of the computational cost.
This content-adaptive strategy is highly effective across different modalities:
- Audio Processing: Haobo Yue et al. introduce Full-Frequency Dynamic Convolution, which generates unique, content-adaptive kernels for every frequency band to isolate sound events from background interference.
- Image Restoration: Xiaoke Shang et al. employ a dynamic filtering mechanism in feed-forward layers that uses a learnable querying process to selectively gate specific frequency bands, optimizing detail restoration and color consistency.
- Multi-Task Adaptation: Xingyu Jiang et al. apply dynamic convolutions within the frequency domain to adaptively filter signal components based on input characteristics, allowing a single model to isolate and remove global degradation patterns across diverse restoration tasks.
Why It Works: Noise Suppression and Global Context
The primary intuition behind learnable frequency filtering is its ability to elegantly solve two major problems in deep learning: separating signal from noise and capturing global context efficiently.
Adaptive Denoising without Loss of Detail
In many sequential and spatial datasets, high-frequency components are dominated by stochastic noise, while low-frequency components contain stable structural patterns. However, rigidly cutting off high frequencies can destroy crucial short-term details or sharp edges.
Emadeldeen Eldele et al. address this by integrating learnable spectral thresholding to dynamically filter out high-frequency noise from time series data without sacrificing critical underlying patterns. For non-stationary data, Wei Fan et al. use adaptive frequency filtering to selectively mask the high-frequency noise that typically gets amplified during mathematical differentiation.
Replacing Self-Attention with Spectral Mixing
Because an element-wise multiplication in the frequency domain updates every token or pixel simultaneously, it acts as a global token mixer. Kun Zhou 0002 et al. exploit this property to completely replace heavy self-attention mechanisms in sequential recommendation models with an all-MLP architecture featuring learnable frequency-domain filters. This reduces the sequence-level time complexity from $O(N^2)$ to $O(N \log N)$ while naturally prioritizing stable behavioral patterns over noisy, transient actions.
Spatial Domain (Self-Attention):
[Token 1] <═══ All-to-All Interactions (O(N²)) ═══> [Token N]
Frequency Domain (Spectral Filtering):
[Tokens] ──> [ FFT ] ──> [ Element-wise Learnable Mask (O(N log N)) ] ──> [ iFFT ]
Clean Disentanglement
Traditional spatial filters often suffer from "spectral leakage," where information from different patterns bleeds together. In contrast, Zhongde An et al. apply learnable masks directly to the frequency spectrum to cleanly separate overlapping trend and seasonal components, avoiding the artificial noise introduced by fixed-window spatial moving averages.
Beyond Raw Signals: Filtering Weights and Graphs
The utility of learnable spectral modulation extends beyond raw input signals like audio and images; it can also be applied to internal model representations, network weights, and non-Euclidean structures.
Spectral Modulation of Model Weights
Instead of filtering data, Peng Sun et al. treat the weight matrices of pre-trained Large Language Models (LLMs) as 2D signals. By applying spectral modulation to selectively dampen high-frequency components within the weight matrices, they remove artifacts introduced during training and reduce parameter noise without requiring any fine-tuning.
Graph Spectral Filtering
In graph neural networks, defining spatial convolutions is challenging due to irregular node connections. Jiayu Fang et al. bypass this by using attention-based modulation of singular values to learn adaptive spectral filters on graphs. This mechanism allows the model to dynamically identify and suppress high-frequency structural noise or amplify specific connectivity patterns based on the unique characteristics of the graph topology.
By shifting operations to the frequency domain and making the filtering process fully differentiable, these techniques provide a mathematically grounded, computationally efficient framework for capturing global context and suppressing noise across diverse machine learning domains.