Neural networks are naturally "lazy" when it comes to learning fine details. If you train a standard network to represent an image or solve a physical equation, it will first capture the broad, smooth shapes before eventually—and often very slowly—filling in the sharp edges and high-frequency textures. This phenomenon is known as spectral bias.
In recent years, researchers have moved beyond simply observing this bias to actively engineering it away. By analyzing the Neural Tangent Kernel (NTK), they have discovered that the speed at which a network learns different frequencies is directly proportional to the eigenvalues of its kernel. To capture fine details, we must "flatten" the NTK spectrum, ensuring that the eigenvalues associated with high-frequency components are not dwarfed by those of low-frequency ones.
The Eigenvalue Learning Rate
To understand why networks struggle with detail, we look at the NTK, a mathematical object that describes how network outputs change during gradient descent. In the infinite-width limit, the training of a neural network behaves like a linear system where the learning speed of different data components is governed by the eigenvalues of the NTK matrix.
Large eigenvalues correspond to fast learning, while small eigenvalues correspond to slow learning. In standard architectures, the eigenvalues decay at an exponential rate. This means the "low-frequency" components (the broad strokes) have massive eigenvalues and converge almost instantly, while "high-frequency" components (the fine details) have tiny eigenvalues, requiring an impractical number of iterations to learn.
Amnon Geifman and colleagues in Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrum demonstrate that by manually reshaping this spectrum to decay more slowly, training time for complex details can be reduced from exponential to constant time.
Architectural Equalizers
One way to flatten the spectrum is to change the network's "wiring." Standard Multi-Layer Perceptrons (MLPs) are additive, but adding multiplicative interactions—such as Hadamard products—fundamentally changes the kernel's decay rate.
Grigorios Chrysos and Moulik Choraria show in The Spectral Bias of Polynomial Neural Networks that polynomial interactions lead to a much slower eigenvalue decay. This allows the model to capture high-frequency information significantly faster than additive models. Similarly, Zhicheng Cai proposed a "split-layer" architecture in Split-Layer: Enhancing Implicit Neural Representation by Maximizing the Dimensionality of Feature Space, which uses parallel branches and Hadamard products to distribute eigenvalues more evenly, effectively mitigating bias without adding parameters.
Other researchers focus on the "floor" of the spectrum. Samuel Audia and Dinesh Manocha proved that Multigrid Parametric Encodings (MPEs) work by raising the minimum eigenvalues of the kernel. By ensuring no eigenvalue is too small, they guarantee that even the finest scales are optimized within a practical timeframe.
Hacking the Training Dynamics
If we cannot change the architecture, we can change how we update it. Several techniques "rebalance" the gradients during training to compensate for the uneven NTK spectrum.
- Inductive Gradient Adjustment (IGA): Kexuan Shi and Shuhang Gu introduced IGA, which uses NTK theory to identify learning speed disparities and applies a transformation matrix to gradients, forcing the model to prioritize high-frequency textures.
- Random Weight Factorization (RWF): Hanwen Wang and Paris Perdikaris found that factorizing weights into scale and direction effectively gives every neuron its own adaptive learning rate, which flattens the NTK spectrum and accelerates convergence in physics-informed neural networks (PINNs).
- Spectrum-Aware Optimizers: Optimizers like Muon and Shampoo are designed to learn all principal components of data at a uniform rate. Bhavya Vasudeva and Christos Thrampoulidis demonstrate that this spectral design prevents simple patterns from dominating early training, which is crucial for generalizing on imbalanced datasets.
Tunable Kernels and Activations
Finally, the choice of the "basis" for the representation—the activation functions or the kernel family—provides a direct knob for spectral tuning.
In FINER: Flexible Spectral-Bias Tuning in Implicit NEural Representation by Variableperiodic Activation Functions, Zhen Liu and Hao Zhu introduced a variable-periodic function that allows users to tune spectral bias simply by adjusting the bias initialization range. This makes the NTK more "diagonal," reducing the mathematical coupling between different spatial coordinates and allowing high-frequency components to converge independently.
Similarly, Maximilian Weiherer showed that Matérn kernels are superior to standard arc-cosine kernels for 3D reconstruction because their bandwidth and smoothness can be explicitly tuned to control eigenvalue decay, ensuring sharp edges are not smoothed away.
By treating the NTK spectrum as a design target rather than a fixed property, these researchers have turned the "Achilles' heel" of neural networks into a controllable parameter, enabling high-fidelity representations in everything from medical imaging to complex fluid dynamics.