When training deep neural networks to solve partial differential equations (PDEs), practitioners frequently encounter a frustrating phenomenon: the model quickly captures the broad, smooth trends of a system but fails to resolve sharp gradients, rapid oscillations, or fine-grained turbulence. This behavior is not a random failure mode; it is a fundamental mathematical property known as spectral bias or frequency-dependent convergence.
In scientific machine learning (SciML), where neural networks act as continuous physics solvers or operators, spectral bias behaves like an implicit low-pass filter. Understanding why this filter exists—and how to break it—is essential for building high-fidelity neural simulations.
The Intuition: Why Networks Learn Low Frequencies First
To understand why neural networks prioritize low frequencies, we must look at how gradients propagate during training. Imagine trying to fit a highly oscillatory wave using a standard multi-layer perceptron (MLP). At initialization, the network's weights map inputs to relatively smooth, random variations.
When we calculate the loss—such as the residual of a differential equation—low-frequency errors produce broad, spatially coherent gradient signals. When mini-batches are sampled, these coherent signals reinforce one another, leading to rapid updates in the weights. Conversely, high-frequency errors oscillate rapidly across the domain. During standard gradient descent, these positive and negative oscillations tend to cancel each other out across a training batch, yielding tiny net gradient updates for fine-scale details.
This phenomenon can be formalized through the lens of the Neural Tangent Kernel (NTK). As shown by Sifan Wang et al. (2022), the eigenvalues of a network's NTK decay at an exponential rate relative to the frequency of the target function. Because the convergence rate along any given eigen-direction is directly proportional to its corresponding eigenvalue, the network eliminates low-frequency components of the physical solution almost immediately, while high-frequency components require an impractical number of optimization steps to resolve.
Fundamentally, this is an optimization bottleneck rather than an expressivity failure. As De Ryck et al. (2024) demonstrated, the network possesses the theoretical capacity to represent high-frequency details, but standard gradient-based optimization algorithms struggle to guide the parameters to those solutions. Furthermore, simply increasing network depth does not resolve this issue; Mema et al. (2025) observed that deeper architectures can become trapped in unstable saddle points within complex energy landscapes, leaving fine-grained oscillations unresolved.
The Cost in Physical Modeling
In classical computer vision, spectral bias is often a feature rather than a bug, as it prevents networks from overfitting to high-frequency pixel noise. In physical modeling, however, spectral bias is a severe limitation. Physical systems are inherently multiscale, and their most critical behaviors often reside in high-frequency or localized regimes:
- Fluid Dynamics and Turbulence: As noted by Karniadakis et al. (2021), fluid systems transfer energy across multiple spatial scales. Spectral bias prevents neural operators from capturing fine-scale eddies and turbulent structures.
- Sharp Boundaries and Fronts: When tracking evolving boundaries, such as wildfire fronts, models smooth out sharp geometric points (Dabrowski et al., 2022). Similarly, in multiphase systems like boiling, frequency-domain operators struggle to resolve the sharp discontinuities at liquid-vapor interfaces (Hassan et al., 2023).
- Stiff Dynamics: In chaotic or highly constrained systems, forces can spike rapidly over minuscule intervals. Spectral bias causes models to over-smooth these transitions, leading to catastrophic divergence during rare but critical physical events (Liang et al., 2022).
- Activation Functions: Standard smooth activations like Tanh or ReLU inherently favor low-frequency representations, causing conventional Physics-Informed Neural Networks (PINNs) to over-smooth solutions and miss rapid transitions (Zhao et al., 2024).
Strategies for Breaking the Low-Pass Filter
To build neural solvers capable of capturing multiscale physics, researchers have developed methods to alter the network's optimization landscape or explicitly inject high-frequency capacity.
Fourier Feature Mappings
One of the most direct ways to counteract spectral bias is to modify the input space before it reaches the deep layers of the network. By passing raw coordinates through a bank of sinusoidal functions with varying frequencies, the input is projected into a high-dimensional space.
Spatial Coordinates (x, t) ---> [ Fourier Feature Mapping: sin(ωk·x), cos(ωk·x) ] ---> Deep MLP ---> Multiscale Solution
As explored by Fazliani (2025) and Deng et al. (2024), this mapping artificially inflates the higher-frequency eigenvalues of the NTK. It forces the network to treat high-frequency spatial variations with the same optimization priority usually reserved for smooth trends.
Domain Decomposition and Hybrid Architectures
Instead of forcing a single global network to learn every frequency simultaneously, domain decomposition strategies break the problem apart. Moseley et al. (2023) demonstrated that dividing a physical domain into smaller subdomains effectively transforms global high-frequency oscillations into local low-frequency trends within each local network's perspective.
Alternatively, hybrid architectures like those proposed by Cooley et al. (2025) combine global MLPs with localized radial basis functions. The global network captures the smooth, macroscopic physics, while the localized functions handle sharp gradients and fine-scale details.
Optimization and Training Adjustments
Modifying the training loop itself can also mitigate frequency bias. Sankaran et al. (2022) observed that utilizing significantly larger batch sizes stabilizes the optimization process for PINNs, preventing high-frequency gradient components from canceling out and allowing the model to resolve complex solution behaviors.
In operator learning, where models map entire functional spaces, the authors of recent frameworks have turned to multi-stage training loops. For instance, Kong et al. (2025) introduced a two-stage training approach for Fourier Neural Operators (FNOs) to explicitly target and correct the frequency bias that typically degrades high-frequency seismic wavefield simulations.
Key Foundational Work
One-Shot Transfer Learning of Physics-Informed Neural Networks (2021) — contributed 1 concept elements to this direction.
Physics-informed machine learning (2021) — contributed 1 concept elements to this direction.