Mitigating Spectral Bias in Neural Networks

82 concepts60 papers1 proposal

Deep neural networks possess a well-documented inductive bias known as spectral bias or the F-principle: they naturally prioritize learning the low-frequency, smooth components of a target function before capturing high-frequency, fine-grained details.

This behavior can be understood mathematically through the lens of the Neural Tangent Kernel (NTK) and the Hessian matrix. During standard first-order optimization (like Gradient Descent), the learning dynamics are governed by the eigenvalue spectrum of the network's kernel. In standard architectures, these eigenvalues decay rapidly. Because the components of the loss gradient corresponding to low-frequency eigenfunctions have much larger eigenvalues, the network rapidly fits the global structure while leaving high-frequency details—such as sharp edges, intricate textures, or highly oscillatory physical boundaries—unlearned for thousands of iterations.

To overcome this bottleneck, a major research direction focuses on actively modulating, balancing, or inverting this spectral bias. This is achieved through three primary mechanisms: reshaping the optimization landscape, engineering frequency-aware architectures, and designing frequency-domain supervision.


Reshaping the Optimization Landscape via Kernel Flattening

If spectral bias is driven by an uneven eigenvalue spectrum in the training kernel, one direct solution is to use optimization techniques that flatten or whiten this spectrum. By equalizing the learning speeds across different frequency components, models can capture fine details and smooth structures simultaneously.

Second-Order and Preconditioned Optimization

First-order optimizers treat all gradient directions uniformly, allowing dominant low-frequency eigenvalues to dictate the pace of training. In a comprehensive study of scientific machine learning optimization, Alena Kopaničáková demonstrates that second-order optimization methods mitigate spectral bias by flattening the spectrum of the training kernel. This narrows the gap between the fastest- and slowest-learning components, preventing the model from exclusively focusing on smooth solutions.

Bonfanti et al. show that Hessian-based optimization acts as a frequency-agnostic training strategy. By using the Hessian to normalize learning rates across the spectral components of the target function, it prevents the model from prioritizing low-frequency approximations, enabling the capture of sharp transitions that first-order optimizers typically fail to resolve. Similarly, Wang et al. evaluate layer-wise Hessian matrices to show that architectures maintaining well-conditioned Hessian eigenvalues across frequencies successfully avoid low-frequency bias.

Preconditioning offers a computationally practical way to achieve this spectral balancing:

  • Gramian Preconditioning: Best McKay et al. introduce Sketchy Natural Energy Gradient Descent (SNGD), which preconditions gradients with an approximated Gram matrix. This reshapes the loss landscape to ensure that low-frequency and high-frequency components are optimized at comparable rates.
  • Curvature Whitening: Jiang et al. demonstrate that curvature-based preconditioning effectively whitens the error across different frequency components. This allows the network to learn all frequencies simultaneously, drastically reducing the generalization delay often associated with "grokking."
  • Spectrum-Aware Optimizers: Beyond standard preconditioning, spectrum-aware optimizers like Muon and Shampoo ensure that all principal components of the data are learned at a uniform rate. As shown by Vasudeva et al., this uniform learning rate prevents simple or frequent patterns from dominating the early stages of training, boosting performance on imbalanced datasets.

Implicit Flattening via Reparameterization

Instead of altering the optimizer itself, the network's internal parameters can be factorized to implicitly alter the NTK spectrum. Wang et al. introduce Random Weight Factorization (RWF), which splits weight matrices into separate scale and direction components. This simple structural change flattens the eigenvalue spectrum of the NTK, allowing coordinate-based networks to learn high-frequency details and low-frequency components at more comparable speeds.

Alternatively, gradients can be adjusted explicitly during the backward pass based on kernel theory. Shi et al. propose Inductive Gradient Adjustment (IGA), which uses NTK eigenvalues to identify uneven learning speeds across frequencies and applies a transformation matrix to the gradients, flattening the learning dynamics without adding inference overhead.


Frequency-Aware Architectures and Input Transformations

Another approach to mitigating spectral bias is to alter the network's architecture or input space, raising the "floor" of learnable frequencies or explicitly routing high-frequency information through dedicated pathways.

Elevating the Kernel Floor

Standard Multi-Layer Perceptrons (MLPs) with ReLU activations exhibit rapid kernel decay. However, Bietti et al. show that the spectral power of neural tangent kernels decays at a slower rate in more complex or convolutional architectures, indicating an inherently superior ability to capture high-frequency details.

Input transformations can directly modulate these kernel properties. Wang et al. demonstrate that applying specific input transformations, such as Fourier feature mappings, modulates the eigenvalues and eigenvectors of the NTK, increasing the network's sensitivity to high-frequency components. Mema et al. leverage this property in variational problems to ensure that microstructural oscillations are not suppressed by the network's natural bias.

To push past the limits of standard Fourier features, Audia et al. prove that Multigrid Parametric Encodings (MPEs) elevate the minimum eigenvalues of a network's kernel function. By raising this eigenvalue floor, MPEs prevent the vanishingly slow optimization of fine-scale features, ensuring high-frequency components are learned within a practical number of iterations.

Standard MLP:      Input ──> [ Rapid NTK Decay ] ──> Low-Frequency Bias (Blurry)
Fourier/Grid:      Input ──> [ Elevated NTK Floor ] ──> Uniform Frequency Learning (Sharp)

Structural Layer Splitting and Reparameterization

Architectures can also be re-engineered internally to expand their feature space. Cai et al. develop a split-layer architecture that distributes NTK eigenvalues more evenly through multi-branch polynomial mappings, enabling implicit neural representations to capture sharp edges without adding extra parameters. Similarly, the authors of an alternative reparameterization framework equalize the learning speed of different frequency components by training MLP weights as a weighted combination of fixed Fourier bases, flattening the NTK eigenvalue distribution during training (Improved Implicit Neural Representation with Fourier Reparameterized Training).

Explicit High-Frequency Sub-bands and Filters

Some frameworks bypass the kernel bottleneck by explicitly separating frequencies structurally:

  • High-Pass Filtering: Yao et al. integrate adaptive high-pass filters directly into the network architecture, forcing the model to prioritize the restoration of sharp edges and textures during the learning process.
  • Wavelet Decomposition: Kim et al. introduce SNeRV, an architecture that explicitly reconstructs high-frequency sub-bands through dedicated modules using wavelet transforms. Yu et al. similarly combine wavelet-based encoders with adaptive harmonic activation functions to preserve fine textures in video compression.

Frequency-Domain Supervision and Loss Priors

When spatial loss functions (like $L_1$ or $L_2$) are minimized, they are naturally dominated by large-scale, low-frequency structural errors. A third strategy introduces loss functions that explicitly penalize errors in the frequency domain, forcing the optimizer to prioritize fine-grained details.

Dynamic and Weighted Frequency Losses

Hu et al. combine spatial feature extraction with explicit frequency-domain supervision, applying a dynamic loss that penalizes errors in high-frequency components to overcome the F-principle. To make this process adaptive, Dai et al. propose Focal Frequency Loss (FFL), which uses dynamic frequency-wise importance weighting to scale the loss at each spectral coordinate based on the current reconstruction error, forcing the network to focus on hard-to-synthesize frequencies.

Spatial Loss Only:     High-frequency errors are mathematically dwarfed by low-frequency structures.
Frequency-Aware Loss:  Weights are dynamically boosted at high-frequency coordinates to compel optimization.

Other applications use static or learnable priors to enforce high-frequency fidelity:

  • Spectral Weight Matrices: Zhang et al. employ a frequency-weighted loss function with a spectral weight matrix to adaptively increase the penalty for errors in high-frequency details, capturing sharp transitions in environmental data.
  • Learnable Loss Priors: Luo et al. introduce a learnable, frequency-aware loss prior that uses spectral resizing to identify high-frequency regions where the model struggles, dynamically adjusting training weights to counteract spectral bias.
  • Transformed Domain Supervision: Cho et al. supervise networks within a transformed frequency domain to force the model to learn specific frequency bands essential for resolving fine geometric structures in sparse-data scenarios.
  • Auxiliary Operators: Li et al. introduce an auxiliary loss term that specifically penalizes errors in the high-frequency domain to prevent neural operators from smoothing out sharp gradients in physical simulations. Tang et al. similarly integrate a frequency-domain loss function via Fourier transforms to preserve sharp edges in compact 3D representations.

Sobolev and Lipschitz Regularization

Rather than penalizing raw frequency coefficients, optimization can be guided by penalizing derivatives or smoothness metrics. Yu et al. show that by applying a sufficiently high penalty to high-frequency errors within a tunable Sobolev-norm loss function, the natural spectral bias of neural networks can actually be inverted, forcing the model to resolve high-frequency patterns before fitting smooth components.

In sequence modeling, the authors of Data Continuity Matters: Improving Sequence Modeling with Lipschitz Regularizer penalize the smoothness (Lipschitz constant) of internal representations, which counteracts spectral bias and accelerates the learning of high-frequency features. Conversely, for systems where high-frequency parameters decay quadratically and cause optimization instability, Yu et al. use Sobolev-norm filtering and targeted initialization to tune the frequency bias of State Space Models, ensuring fine-grained details are not lost.


Data Transformations and Curriculum Strategies

A final facet of this research direction manipulates the training data itself or controls the schedule by which the network encounters different frequencies.

Disrupting Low-Frequency Structures

If a network naturally latches onto smooth global structures, removing those structures forces the network to adapt. Seo et al. introduce a counterintuitive data transformation called Random Pixel Permutation (RPP). By intentionally destroying low-frequency spatial patterns in a target signal, RPP forces the network to treat all data points as equally difficult, preventing spectral bias from prioritizing smooth regions and enabling a rapid convergence phase for high-frequency details.

Curriculum Learning and Refinement

Instead of forcing simultaneous learning, models can be guided through a frequency-staged schedule:

  • Frequency Curriculums: Meng et al. propose a frequency-based curriculum learning strategy that trains the model to master low-frequency components before progressively injecting higher-frequency information, mitigating the optimization difficulties of sharp textures.
  • Coarse-to-Fine Refinement: Kim et al. introduce a "zoom-in, refine, and zoom-out" strategy for image completion. By upscaling coarse results and refining them at a higher resolution with high-frequency supervision, the framework reconstructs sharp details that standard models smooth away.

Tuning Bias for Robustness

While most methods seek to maximize high-frequency learning, certain tasks require dampening it to prevent overfitting to high-frequency noise. Chan et al. develop Jacobian Frequency Regularization (JaFR), using a hyperparameter to explicitly steer a model's reliance toward low frequencies to match natural image statistics, significantly improving adversarial robustness. Conversely, for open-set recognition, Lijun Liu introduces a framework that decouples texture and structure by shuffling high-frequency details while enhancing low-frequency global structures, correcting the network's tendency to over-rely on spurious high-frequency noise.

Key Foundational Work

Zoom-to-Inpaint: Image Inpainting with High-Frequency Details (2020) — contributed 1 concept elements to this direction.
Frequency Bias in Neural Networks for Input of Non-Uniform Density (2020) — contributed 1 concept elements to this direction.
On the Inductive Bias of Neural Tangent Kernels (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_spectral_bias_mitigation_high_frequency__ecaf73bf,
  title = {spectral bias mitigation (high-frequency feature learning)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spectral-bias-mitigation-high-frequency-feature-learning/rdr_ecaf73bfd4e84c59ad54736dda57b264},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_spectral_bias_mitigation_high_frequency__ecaf73bf_v1,
  title = {spectral bias mitigation (high-frequency feature learning)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spectral-bias-mitigation-high-frequency-feature-learning/rdr_ecaf73bfd4e84c59ad54736dda57b264/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake