Band-wise Step Sizes for Continual Neural Fields

A neural field is a small network that stores a signal, such as an image, a video or a 3D scene, by mapping coordinates like $(x, y, t)$ to values like color. When such a network is trained on data that arrives over time, two problems collide. The first is spectral bias: gradient descent fits the smooth, low-frequency parts of a signal quickly and the fine, high-frequency details very slowly. The second is the classic learning-rate dilemma of continual learning. A large step size lets the network absorb new data but overwrites what it already stores. A small one protects old content but leaves the network unable to learn. Today both problems are handled with one knob, a single global learning rate, or with a coarse static split of parameters into "fast" and "slow" groups. We argue this is the wrong granularity. The trade-off between learning and forgetting is set separately in each frequency band, and it pulls in opposite directions in low and high bands.

Our central hypothesis follows from how gradient descent spreads each update across the input domain. When a network updates on new coordinates, the low-frequency part of the update is spatially broad, so it disturbs predictions at every old coordinate. The high-frequency part is narrow, so it mostly changes predictions near where the new data lie. If this holds, forgetting in continual neural fields comes mainly from low bands, while the plasticity bottleneck, the inability to fit new detail, sits in high bands. The two can then be decoupled with separate step sizes per band: small and cautious for broad structure, large and responsive for fine detail. Each step size is adjusted online from cheap, frequency-resolved error and interference signals. We propose to test this diagnosis directly, build the controller, and evaluate it against the strongest fixed-budget alternatives. We also name in advance the static-schedule result that would make the dynamic part unnecessary.

Core Claim

In the linearized, or "neural tangent kernel" (NTK), view of training, a network's function changes along the eigenfunctions $\phi_k$ of a kernel $K$. That kernel measures how an update at one input moves the output at another. Each eigenfunction has an eigenvalue $\lambda_k$, which sets how quickly training moves the function along that direction. On the eigenvector bias of Fourier feature networks showed that spectral bias is exactly this: high-frequency eigenfunctions have near-zero eigenvalues, and they learn slowly. The paper fixed this by reshaping the kernel with Fourier input features, not by changing step sizes. We give band $b$ its own step size $\eta_b$. Then the change caused at old coordinates $x_{\text{old}}$ by one step on a new residual $r_{\text{new}}$ is, to first order,

$$ \Delta f(x_{\text{old}}) = -\sum_{b}\ \eta_b \sum_{k \in b} \lambda_k \,\langle \phi_k, r_{\text{new}} \rangle\, \phi_k(x_{\text{old}}). $$

The same product $\eta_b\lambda_k$ also sets how fast band $b$ fits the new data. So learning speed and forgetting speed are tied within a band but not across bands. The quantity that decides whether a band causes forgetting is the overlap term $\langle \phi_k, r_{\text{new}}\rangle\phi_k(x_{\text{old}})$, the reach of that band's update into old territory. Our empirical claim is that this reach is much larger for low bands when new data are spatially or temporally separated from old data.

If that is right, the "tension" the literature describes, where boosting high-frequency plasticity accelerates forgetting (Plastic Learning with Deep Fourier Features notes its periodic networks overfit so easily they need heavy regularization), is mostly an artifact of boosting every band at once. Boosting only the high bands should raise plasticity at little cost to stability.

Relation to Prior Work

Two papers motivate the approach and supply its parts. Random Weight Factorization rewrites each weight matrix as a per-neuron scale times a direction. It shows this acts as an implicit neuron-level adaptive learning rate that flattens the kernel spectrum, cutting advection-equation error from over 28% to 4.14%. Its adaptivity is static in design and blind to frequency and to forgetting. We make the per-band rate explicit and let it respond to the data stream. Learning Rate Schedules in the Presence of Distribution Shift proves that under drift the optimal rate should rise with shift velocity, and Surprise-based learning uses a surprise signal to raise plasticity. Both control one scalar rate. We borrow their error-driven logic and apply it band by band.

The closest existing work is ChannelWavAct. It names spectral bias as a cause of plasticity loss and trains a "plasticity group" and a "stability group" of parameters at two fixed learning rates. It also injects new wavelet bases when the loss plateaus. Its split is by parameter type (weights versus biases and normalization), not by frequency, and it is static. It adds parameters, where we add none. Online Prototypes and Class-Wise Hypergradients adapts per-parameter rates from the inner products of consecutive gradients. It works on pretrained transformer classifiers with no frequency structure. We keep its idea that the rate should follow the signal, but use frequency-resolved residuals.

Three works constrain the design, because they solve continual neural fields by giving up a fixed budget. PFNR freezes a sparse Fourier subnetwork per video, which gives zero forgetting until capacity runs out. Meta-Continual Learning of Neural Fields adds a module per task, so parameters grow linearly. Instant Continual NeRF uses spatial hash grids for locality and distillation from a frozen copy. We aim to recover part of their stability inside one fixed-size coordinate network, with no stored targets.

Spectral Collapse Drives Loss of Plasticity shows that learning requires error to project onto the fast eigenspace, and that this eigenspace collapses over many tasks. Its "spectrum" is the Hessian and feature covariance, and it regularizes rank through curvature approximations. Our proposal is complementary. We act on step sizes per input-frequency band rather than on representation rank, and we test whether band rates alone slow the collapse it describes.

Research Program

The first stage is diagnostic and does not depend on any new method. We fit a Fourier-feature MLP and a SIREN-style sinusoidal network sequentially on streams whose new data are separated from the old in coordinate space. We then decompose forgetting by frequency: after each task, we take the error on all past coordinates, Fourier-transform it on the known grid, and report the forgetting energy per radial frequency band. For small networks we also compute the empirical NTK on a probe grid and measure each eigen-band's reach into old regions directly. That lets us check the linearized prediction above against the measured forgetting. The hypothesis fails if forgetting energy is spread evenly across bands, or concentrates in high bands.

The second stage builds the controller. To get band-specific step sizes without eigendecomposing a kernel, we use a multi-scale Fourier-feature input in the style of On the eigenvector bias of Fourier feature networks. There, separate branches encode the input at a small number $B$ of frequency scales. Each branch's first-layer weights, and optionally its random-weight-factorization scales, form one band group with its own rate $\eta_b$. The trunk shared after the branches keeps a base rate.

Each $\eta_b$ is set online from two signals. The first is a plasticity signal: the ratio of band $b$'s current residual energy on incoming data to its running average, following the surprise and drift logic of the scalar-rate work. The second is an interference signal: how much a tentative band-$b$ update would change the network's outputs at a fixed set of past probe coordinates. Computing this requires only coordinates and forward passes, never stored targets. The rate grows with the first signal and shrinks with the second, clipped to a range. The controller has two meta-hyperparameters, a base rate and an averaging constant. Both are tuned once and held fixed across benchmarks, to address the hyperparameter burden the file flags for adaptive-rate methods.

The third stage asks whether the effect survives beyond the linearized regime. We track how band rates evolve over long streams, check the controller's stability when rates grow large (where continuous approximations of SGD are known to break down), and measure the effective rank of hidden features to see whether band rates slow the collapse described in Spectral Collapse Drives Loss of Plasticity.

Baselines

The naive reference point is the same multi-scale Fourier-feature MLP fine-tuned sequentially with Adam at one learning rate. It uses the same parameter count, the same number of steps per task and the same total budget for tuning its learning rate. It sets the floor for both forgetting and plasticity. A second weak reference is the same network with random weight factorization, Random Weight Factorization. It tells us whether implicit neuron-level rates already capture the effect.

The strongest existing competitor at a fixed budget is the static two-group scheme of ChannelWavAct. We reimplement its decoupled optimization on our backbone, with or without its wavelet activations, and give it the same total tuning trials as our controller. It is the only prior method that deliberately assigns different rates to "plastic" and "stable" parameters in a spectral-bias setting. We also run a per-parameter hypergradient controller in the style of Online Prototypes and Class-Wise Hypergradients, to separate "dynamic rates" from "frequency-resolved dynamic rates."

The growth-based methods are reported as upper references, not same-budget competitors. On CelebA, Meta-Continual Learning of Neural Fields reports 45.4 dB PSNR with zero forgetting through per-task modules, using 787 MiB of GPU memory. On UVG8, PFNR reports 32.12 dB average PSNR, 0.93 MS-SSIM and exactly zero backward transfer, against 27.66 dB for its WSN baseline. For Instant Continual NeRF the file reports speedups but no PSNR, so no reported fidelity result was found on Lacuna for it. We will compare it only on streaming-reconstruction runs we perform ourselves.

The key ablation is a static band-wise schedule: fixed per-band rates found by grid search with the same tuning budget, and no online signals. A second ablation removes the interference signal, and a third removes the plasticity signal. Beating the baselines means lower average forgetting and higher final average PSNR than both the naive and the two-group baselines at identical parameter count and compute. We would claim the dynamic controller matters only if it also beats the tuned static band schedule, by a margin larger than the spread across seeds.

Why not just…?

Why not just use Adam, which already adapts each parameter's rate? Adam rescales along individual parameter axes by gradient magnitude, not along frequency bands or kernel eigenfunctions. The file reports that standard optimizers still fail completely on multi-scale problems, with 100% error on a multi-scale wave equation in On the eigenvector bias of Fourier feature networks, and still suffer spectral collapse in continual training. Because every baseline and our method already run on Adam, the naive baseline directly measures what Adam alone achieves.

Why not just reweight the loss by frequency instead of changing step sizes? This is a real threat. Up-weighting the high-frequency part of the residual speeds high-band learning in function space, and in the linearized regime it is almost equivalent to raising high-band rates. It cannot, however, reduce how far a low-band update reaches into old coordinates. That reach depends on how the parameters that move the low band are updated, not on how the residual is weighted, and suppressing that reach is the stability half of our claim. We will include a frequency-weighted-loss baseline with the same tuning budget. If it matches our method on forgetting, not just on plasticity, the interference half of the hypothesis is wrong.

Why not just tune one fixed rate per band? If the needed trade-off is stable over a stream, a grid-searched static band schedule should do as well as the controller. We expect it to lose on streams whose frequency content drifts, where the scalar-rate theory in Learning Rate Schedules in the Presence of Distribution Shift predicts that fixed rates accumulate regret. It may well tie on streams with uniform statistics. If it ties everywhere, we will drop the controller and present the result as a diagnosis plus a simple recipe: small rates for low bands, large rates for high bands.

Why not just use hash grids or replay? Spatial hash grids get locality by construction, but they are memory-heavy, and Instant Continual NeRF still needs distillation from a frozen copy. Replay stores targets. Our question is how much of that stability a compact coordinate network can reach with neither. We therefore report memory alongside fidelity, rather than claiming to beat grids on raw PSNR.

Proposed Experiments

We test on three streams. The first is a controlled spatial stream: large natural images revealed tile by tile, which tests the claim that forgetting lives in low bands. The second is sequential CelebA images in the protocol of Meta-Continual Learning of Neural Fields. The third is sequential UVG8 video compilation in the protocol of PFNR. A synthetic drift stream, whose dominant frequency rises over time, isolates the value of the online plasticity signal. We report average PSNR, MS-SSIM on video, backward transfer, forgetting energy per band, parameter count and memory, over at least five seeds.

The first decisive check is the per-band forgetting decomposition under plain fine-tuning. If low bands do not dominate forgetting on the separated tile and CelebA streams, the core claim fails regardless of downstream PSNR. The second is whether the measured NTK band reach predicts the measured per-band forgetting in small networks. The third is the comparison with the tuned static band schedule on the drift stream versus the uniform streams, which decides whether the dynamic controller is warranted. The fourth is how far a single fixed-budget network closes the gap to the growth-based references, 45.4 dB on CelebA and 32.12 dB on UVG8, while staying above the capacity ceiling PFNR reports on long sequences. We will report this gap as it stands, not claim parity.