Introduction
Continuous Neural Representations (CNRs) have revolutionized how we represent complex data in computer vision and scientific computing. By using Multi-Layer Perceptrons (MLPs) to map coordinates (e.g., $x, y, z$) directly to values (e.g., color, density, or physical state), CNRs offer a memory-efficient and differentiable alternative to traditional discrete grids. Despite their success in applications like Neural Radiance Fields (NeRF), these networks suffer from spectral bias—a tendency to learn low-frequency, "blurry" signals before high-frequency details. Overcoming this bias often requires meticulous hyperparameter tuning of specialized architectures like SIREN or Fourier features.
Random Weight Factorization (RWF)
To address these limitations, researchers have introduced Random Weight Factorization (RWF), a simple yet effective "drop-in" modification for linear layers. RWF re-parameterizes the weight matrix $W$ as a product of a trainable scale vector $s$ and a trainable direction matrix $V$: $$W = \text{diag}(s) \cdot V$$ Crucially, the scale factors $s$ are initialized randomly from a log-normal distribution. This factorization changes the geometry of the loss landscape, effectively granting each neuron a self-adaptive learning rate.
As shown in Figure 1, the transformation into scale-direction space can bring the global minimum closer to the initialization point. If a neuron needs to capture a sharp, high-frequency detail, its scale factor $s$ can grow, accelerating its optimization relative to other neurons in the network.
Theoretical Insights: Flattening the Spectrum
The paper provides a rigorous analysis of RWF using the Neural Tangent Kernel (NTK) framework. The NTK describes how a network's predictions evolve during training. A common cause of spectral bias is a highly skewed NTK eigenvalue spectrum, where a few large eigenvalues dominate the learning of low frequencies. RWF is shown to lead to a flatter eigenvalue spectrum, meaning the network can learn a wider range of frequencies at more comparable speeds. This theoretical finding is a "smoking gun" for the reduced spectral bias observed in practice.
Experimental Evaluation
The authors tested RWF across an extensive range of benchmarks, demonstrating its versatility and robustness:
- 2D/3D Representation: RWF achieved superior PSNR in image regression and higher IoU in 3D occupancy networks for complex meshes like the "Dragon."
- Inverse Problems: In 2D Computed Tomography and 3D NeRF rendering, RWF consistently outperformed standard MLPs and Weight Normalization.
- Physics-Informed Neural Networks (PINNs): RWF showed remarkable success in solving Partial Differential Equations (PDEs).
The most striking result was observed in the Advection equation, a notorious failure case for standard PINNs. While "plain" MLPs and other normalization techniques failed to capture the wave dynamics (errors > 28%), RWF achieved a relative $L^2$ error of just 4.14%, as visualized in Figure 6.
Robustness and Efficiency
A key advantage of RWF is its robustness. Unlike methods that require precise tuning of scale hyperparameters, RWF works well with a default initialization across diverse tasks. Furthermore, it introduces negligible computational overhead, maintaining nearly the same training speed as standard MLPs.
Conclusion
Random Weight Factorization provides a powerful and easy-to-implement solution for the spectral bias inherent in coordinate-based neural networks. By reshaping the optimization landscape and enabling self-adaptive learning at the neuron level, RWF allows models to capture intricate details that were previously difficult to reach. Future research may extend this factorization to other architectures, such as Transformers or CNNs, to see if similar benefits apply beyond the domain of continuous representations.
Summary updated