TL;DR
Training continuous neural representations (coordinate-based MLPs) often stalls due to spectral bias and poor local minima. This paper introduces Random Weight Factorization (RWF), a lightweight re-parameterization that factors each neuron’s weight vector into a scalar scale factor and a direction vector, initialized randomly. This simple drop-in modification transforms the loss landscape, provides per-neuron adaptive learning rates, and yields dramatic improvements across image fitting, 3D shape modeling, neural radiance fields (NeRF), and physics-informed neural networks (PINNs).
What Problem Does This Paper Address?
Coordinate-based MLPs map spatial or spatio-temporal coordinates $(x, y, z, t)$ directly to continuous signals like RGB colors, 3D occupancy fields, or physical quantities in differential equations. However, standard MLPs suffer from spectral bias: they learn low frequencies quickly while failing to capture high-frequency details and fine spatial features.
Existing remedies—such as Fourier feature encodings or periodic activation networks (like SIREN)—often demand sensitive hyperparameter tuning (e.g., coordinate scale factors). If tuned improperly, they cause severe aliasing or training divergence. Furthermore, self-supervised formulations like PINNs easily get trapped in non-physical local minima near their initialization.
Methods
The authors propose factorizing the weight vector $\mathbf{w}^{(k,l)} \in \mathbb{R}^{d_{l-1}}$ of the $k$-th neuron in layer $l$ as: $$\mathbf{w}^{(k,l)} = s^{(k,l)} \cdot \mathbf{v}^{(k,l)}$$ or in matrix form, $W^{(l)} = \text{diag}(\exp(\mathbf{s}^{(l)})) \cdot V^{(l)}$.
What is new vs. standard:
- Initialization: Instead of setting scale factors to $1$ (like standard adaptive activations) or fixing them by weight norms (like standard Weight Normalization), $\mathbf{s}^{(l)}$ is sampled from a multivariate normal distribution $\mathcal{N}(\mu, \sigma I)$ (recommended: $\mu = 1.0, \sigma = 0.1$). Weights are then initialized as $V^{(l)} = \text{diag}(\exp(-\mathbf{s}^{(l)})) W^{(l)}_{\text{Glorot}}$.
- Exponential Parameterization: Enforces strictly non-zero scales and spans multiple orders of magnitude.
- Drop-in Implementation: Adds only $\mathcal{O}(L \cdot D)$ parameters (one scalar per neuron) with essentially zero computational overhead.
# Flax implementation of RWF linear layer
kernel = jnp.exp(s) * v
y = jnp.dot(x, kernel) + bias
Datasets
The authors evaluated RWF on diverse benchmarks spanning vision, graphics, and scientific machine learning:
- 2D Image Regression: Center crops of 16 images from Div2K (Natural) and 16 procedurally generated Text images (downsampled $2\times$ for training, evaluated at full $512 \times 512$).
- 3D Shape Representation: Standard Stanford 3D triangle meshes (Dragon and Armadillo) tested near the surface boundary via Occupancy Networks.
- 2D Computed Tomography (CT): Procedural Shepp-Logan phantoms and 20 clinical MRI brain scans from the ATLAS dataset.
- 3D Inverse Rendering: Synthetic NeRF Lego dataset (100 train / 7 val / 13 test views).
- Partial Differential Equations (PINNs): 1D Advection ($c=50$) and 2D Lid-driven cavity Navier–Stokes ($Re=1{,}000$).
- Learning Operators (DeepONets): Nonlinear Diffusion-Reaction PDE, Darcy flow through porous media, and 1D viscous Burgers’ equation.
Key Insights and Ideas
- Geometric Landscape Compression: In the factorized $(s, v)$ space, every weight vector maps to a hyperbola $w = s \cdot v$. As $s$ grows, the Euclidean distance between any two parameter states—such as initialization and the global minimum—shrinks toward zero (Theorem 1).
- Self-Adaptive Neuron Learning Rates: Projected back into original parameter space, the gradient update scales by $\left([s^{(k,l)}]^2 + \|\mathbf{v}^{(k,l)}\|_2^2\right) \frac{\partial \mathcal{L}}{\partial \mathbf{w}^{(k,l)}}$ (Theorem 2). Because $s$ and $v$ evolve dynamically, each neuron learns at its own self-adjusting rate.
- Spectral Bias Mitigation: Computing the empirical Neural Tangent Kernel (NTK) shows that RWF produces a significantly flatter eigenvalue spectrum with slower decay, preventing high frequencies from vanishing during training.
- Rethinking SIREN: Analysis reveals that SIREN's performance gains stem primarily from its hidden-layer scale factor $\omega_0$ (which behaves as a fixed weight factorization) rather than the periodic sine activation itself.
Key Results
RWF consistently outperformed standard MLPs, Adaptive Activations (AA), and Weight Normalization (WN) under identical budgets and learning schedules:
| Task / Benchmark | Metric | Plain MLP | Weight Norm (WN) | RWF (Ours) |
|---|---|---|---|---|
| Image Regression (Natural) | PSNR (↑) | 27.35 dB | 27.36 dB | 28.08 dB |
| Image Regression (Text) | PSNR (↑) | 32.09 dB | 32.25 dB | 33.13 dB |
| 2D CT (Shepp-Logan) | PSNR (↑) | 30.09 dB | 30.59 dB | 33.73 dB |
| 2D CT (ATLAS Brain) | PSNR (↑) | 21.85 dB | 22.16 dB | 23.71 dB |
| PINN: Advection ($c=50$) | Rel. $L^2$ (↓) | 25.94% | 67.40% | 3.10% |
| PINN: Navier-Stokes ($Re=1000$) | Rel. $L^2$ (↓) | 11.93% | 14.62% | 6.79% |
| DeepONet: Diffusion-Reaction | Rel. $L^2$ (↓) | 1.09% | 0.97% | 0.50% |
| DeepONet: Burgers' Equation | Rel. $L^2$ (↓) | 5.11% | 4.37% | 2.46% |
On hard PINN problems like high-speed advection, baselines completely failed to learn the wave pattern, while RWF solved the system accurately (dropping further to 2.08% when combined with causal training).
Limitations and Future Work
- Initialization Hyperparameters: Performance depends on the distribution parameters $(\mu, \sigma)$ of the initial scale factor $s$. An ablation shows a sweet spot around $\mu \in [1, 2]$ and $\sigma \in [0.01, 0.1]$; values set too large cause numerical instability.
- Scope: The evaluation focuses primarily on coordinate-based MLPs and operator networks. Extending RWF to architectures like CNNs, Graph Neural Networks, and Transformers remains open for exploration.
Summary updated