Spectral Bias in Coordinate Neural Networks

1,602 concepts921 papers7 proposals

When training a Multilayer Perceptron (MLP) to map low-dimensional coordinates—such as $(x, y)$ pixel locations or $(x, y, z)$ spatial points—directly to signal values like color, density, or depth, a curious phenomenon occurs. Instead of fitting the fine details immediately, the network first outputs a highly smoothed, blurry approximation of the target signal.

This phenomenon is known as spectral bias (or frequency-dependent convergence). It describes the inherent mathematical tendency of neural networks to prioritize learning low-frequency components (broad shapes and global structures) before capturing high-frequency details (sharp edges, fine textures, and rapid oscillations).

As Mildenhall et al. observed during the development of Neural Radiance Fields (NeRF), standard MLPs struggle to represent sharp edges and intricate geometric details because of this low-frequency preference. Whether reconstructing a 3D scene or performing image super-resolution, as explored by Chen et al., the network acts as an unintentional low-pass filter during the early stages of optimization.

Why Networks Learn Low Frequencies First

To understand why spectral bias happens, we must look at how gradient descent updates network weights. When an MLP is fed raw, low-dimensional coordinates, the gradients corresponding to low-frequency variations are significantly larger and more coherent than those for high-frequency variations. During backpropagation, the network naturally takes large steps toward resolving the coarse structure of the data, while the fine-grained details are left as negligible residual errors to be solved much later.

This optimization imbalance is compounded by the statistical nature of real-world data. As Ballé et al. point out, natural images exhibit a $1/f$ power spectrum, meaning signal energy is heavily concentrated in low-frequency bands. When a network attempts to minimize reconstruction error, the dominant low-frequency signals overshadow the subtle high-frequency details, leading to slow convergence for fine-grained features.

Furthermore, the choice of activation function plays a critical role. Standard rectified linear unit (ReLU) networks possess a mathematical smoothness constraint that inherently biases them toward low-frequency functions. Wang et al. analyzed this through the lens of the network's eigenvector bias, demonstrating that standard architectures struggle to resolve multi-scale features or highly oscillatory patterns. This limitation severely hinders physics-informed neural networks (PINNs) when solving complex differential equations with rapid variations.

The Pervasive Cost of Smoothness

Spectral bias is not confined to computer vision; it is a fundamental characteristic of deep learning that spans multiple domains:

  • Geometric and Structural Loss: In font generation, standard implicit representations round off sharp corners and lose structural fidelity because the network prefers smooth transitions (Reddy et al.). Similarly, in modeling human avatars, Zheng et al. observed that coordinate MLPs smooth away fine garment wrinkles and fabric textures.
  • Medical and Geophysical Imaging: In sparse-view computed tomography (CT) and parallel MRI, spectral bias prevents the recovery of fine anatomical structures and sharp boundaries essential for diagnostic accuracy (Zang et al.; Wu et al.). In geophysics, Goyes-Peñafiel et al. found that coordinate networks struggle to interpolate missing seismic wavefields accurately because they fail to capture high-frequency structural details.
  • Signal Mapping: Even in wireless relay networks, Kalogerias demonstrates that standard MLPs fail to map the sharp peaks and valleys of spatial signal strength due to this low-frequency preference.

Overcoming the Bias

To force neural networks to learn high-frequency details, researchers have developed several strategies that alter the input space, the activation functions, or the network factorization.

Input Transformation (Fourier Features and Splines)

The most common solution is to map low-dimensional coordinates into a higher-dimensional space using periodic functions before passing them to the MLP. By preprocessing coordinates with a bank of sinusoids at varying frequencies (Fourier features), the network's effective kernel bandwidth is altered. This mapping "stretches" the coordinate space, allowing the network to treat high-frequency details with the same optimization priority as low frequencies.

Alternatively, Wang et al. introduced Spline Positional Encoding (SPE), replacing fixed sinusoids with trainable B-splines to recover intricate geometric details without introducing the noisy artifacts or "ghosting" common to high-frequency Fourier mappings.

Periodic Activation Functions

Instead of changing the inputs, another approach changes the network's internal activations. Chan et al. leveraged periodic sinusoidal activation functions (SIREN) to synthesize high-quality, 3D-consistent images with fine textures. Because the derivative of a sine wave is another sine wave, these networks preserve high-frequency information across layers, bypassing the smoothing limitations of ReLU networks entirely.

Weight Factorization

Rather than changing the architecture or inputs, Wang et al. proposed Random Weight Factorization (RWF). By splitting weights into scale and direction components, they effectively provide adaptive learning rates per neuron. This simple architectural adjustment accelerates the convergence of high-frequency details, allowing networks to capture fine structures without requiring complex coordinate transformations.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_spectral_bias_frequency_dependent_conver_c4e27b3d,
  title = {spectral bias (frequency-dependent convergence)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spectral-bias-frequency-dependent-convergence/rdr_c4e27b3d34f14ef1afae3c10e2e3a010},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_spectral_bias_frequency_dependent_conver_c4e27b3d_v1,
  title = {spectral bias (frequency-dependent convergence)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spectral-bias-frequency-dependent-convergence/rdr_c4e27b3d34f14ef1afae3c10e2e3a010/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake