Data-Aware Spectral Repair: Fixing Collapsed Weight Directions Only Where the Data Lives

Networks that keep training on a changing stream of tasks gradually lose the ability to learn new ones. A clear account of why comes from SingularClip. As training proceeds, each weight matrix concentrates its energy in a few directions. The smallest singular values (the gains the matrix applies along its weakest directions) collapse toward zero, and gradients along those directions shrink with them. SingularClip's fix is to take an exact singular value decomposition (SVD) of every weight matrix every $K$ steps and clip all singular values into a fixed band. The fix works, but it has two costs. It pays a cubic-cost factorization per layer. It also rewrites the matrix along every direction, including directions the current data never touches and that may hold old knowledge.

We propose a narrower repair. Keep a running estimate of the few directions that the layer's incoming activations actually occupy. This is done with online subspace tracking, a streaming PCA method that updates a small orthonormal basis from each minibatch at cost linear in the layer width. Then measure and repair the weight matrix's conditioning only inside that tracked subspace. The repair needs matrix multiplications, one thin QR, and a $k \times k$ SVD, with no full factorization. Our central bet is not mainly about speed. It is that conditioning the repair on the data stream gives a better trade between keeping plasticity and preserving what was learned than clipping everything. The speed gain is a secondary benefit.

Core Claim

Let $W \in \mathbb{R}^{m \times p}$ be a layer's weight matrix and $U_t \in \mathbb{R}^{p \times k}$ an orthonormal basis for the top-$k$ subspace of the layer's input activations at step $t$. The plasticity-relevant quantity is the conditioning of $W$ restricted to $U_t$, meaning the singular values of $W U_t$. The conditioning of $W$ over all of $\mathbb{R}^p$ is the wrong target.

The repair works as follows. Write $W U_t = Q R$ (thin QR, with $Q \in \mathbb{R}^{m \times k}$) and $R = P S V^\top$ (a $k \times k$ SVD). Then set

$$ W \leftarrow W + Q P \big(\mathrm{clip}_{[\bar\sigma/c,\; c\bar\sigma]}(S) - S\big) V^\top U_t^\top , $$

where $\bar\sigma$ is the median of the diagonal of $S$ and $c$ is the band width. After this update, $W U_t$ has its singular values clipped. $W$ is left exactly unchanged on the orthogonal complement of $U_t$. Per-layer cost is $O(mpk)$ for the product and $O(mk^2)$ for the QR, compared with $O(mp\min(m,p))$ for exact SVD.

The claim is falsifiable on two fronts. First, on SingularClip's own plasticity benchmarks, the restricted repair should match exact full clipping on plasticity. Second, it should show measurably less forgetting on a stream with returning tasks, because it leaves directions the current data does not excite untouched.

Research Program

The tracker is the block stochastic power method from Memory Limited, Streaming PCA. It multiplies the current basis by each minibatch's activation covariance and re-orthonormalizes with QR, keeping an $O(pk)$ footprint. The single-vector Oja analysis in Near-optimal stochastic approximation for online principal component estimation and Matching Matrix Bernstein with Little Memory shows that such trackers match batch PCA's statistical error under stationarity. We supply the method from these papers and change one thing. We use a constant step size (exponential forgetting) so the basis follows task switches, since the file notes abrupt shift as the standard failure of streaming PCA.

We treat tracking lag as a quantity to measure, not an assumption. After each task switch we record the principal angle between $U_t$ and a batch-SVD basis of the new task's activations. We relate repair quality to that angle.

JANUS-LORA is the closest prior use of in-loop tracking. It tracks a historical activation subspace on the Stiefel manifold, but only to forbid updates inside it, which is the rigid strict-orthogonality stance. LoDA and ROGO show that stance hurts forward transfer. Our use is the opposite. The tracked subspace marks where the network must stay responsive, so its gains there are kept well conditioned. It is not a region to wall off.

QGPM and Share also reduce SVD cost, through sketching and periodic batch SVD respectively. Both serve projection memories or merging rather than spectrum repair.

We also use the restricted condition number, the ratio of largest to smallest singular value of $W U_t$, as a cheap diagnostic in its own right. We ask whether a representation ceiling is spectral. The no-go result in Is PEFT Enough for Cell Segmentation? motivates this. On LIVECell, base Cellpose-SAM, every adapter variant, and full fine-tuning all land near 0.56 AP. We will compute restricted condition numbers of Cellpose-SAM layers on LIVECell activations. If they are unremarkable, the diagnostic tells us that ceiling is not a collapse of the AIPL type and that no spectral method should be expected to move it. That result would constrain our claims. We will report it either way.

Baselines

The naive reference is plain training with no spectral intervention, the same optimizer, and the same schedule. It shows how much plasticity is lost at all.

The strongest competitor on the plasticity question is exact SingularClip SingularClip: full SVD every $K$ steps with $c \in \{2, 4\}$. We run it with the authors' $K$ and $c$ grid, on the same architectures, seeds, and data order. It gets the same hyperparameter budget as our method. SingularClip's results are reported only graphically in the file, so for its benchmarks there is no reported result found on Lacuna for SingularClip as a scalar. We will rerun it ourselves.

We also include periodic weight resetting. SingularClip reports that resetting fails on DMC Humanoid-Stand, and including it keeps the comparison honest. We also include one-sided spectral-norm regularization, which the file indicates leaves $\sigma_{\min}$ unconstrained.

For the continual fine-tuning setting with a pretrained backbone, the reference is Janus-LoRA. It reports 77.11% mean average accuracy on 20-task ImageNet-R JANUS-LORA. We use the same ViT backbone and task split.

Two ablations isolate the key ingredient. The first replaces the tracked $U_t$ with a random orthonormal $k$-dimensional basis, redrawn at the same frequency. If data awareness matters, this should lose. The second replaces the online tracker with an exact batch SVD of buffered activations every $K$ steps. This separates the value of the restricted repair from the value of doing it online.

Beating SingularClip means one of two outcomes. The first is equal plasticity (final-task learning accuracy and per-task learning speed within seed noise) with lower forgetting on returning tasks. The second is equal forgetting at lower cost, measured as per-step wall-clock and peak memory. On ImageNet-R, beating Janus-LoRA means exceeding 77.11% under matched budget. We do not treat that as the primary claim.

Why not just…?

Why not just keep exact SVD and raise $K$ so the cost amortizes away? This is a real threat to any purely computational argument. For ViT-scale widths, an SVD every thousand steps may be affordable. The file's in-loop contrast of 1700 ms versus 20 ms per step in Approximate Muon with low-rank adapters concerns a different operation. That is why our main claim rests on forgetting, not speed. Our experiments sweep $K$ for exact SingularClip. If large-$K$ exact clipping matches our forgetting and plasticity, the proposal reduces to an engineering speedup, and we will say so.

Why not approximate full clipping with matmul-only polynomial iterations, as SingularClip itself suggests with Newton-Schulz? This removes the SVD, but it still rewrites every direction of $W$. That is exactly the behavior we hypothesize causes unnecessary forgetting. We implement a polynomial clip, matched in matmul count to our method, as an additional baseline. The random-subspace ablation and the returning-task forgetting measurement then decide whether restriction to the data subspace is what matters.

Why not track the weight matrix's own weakest singular directions, for example with preconditioned solvers like the RLOBPCG method of Fast, High-Accuracy, Randomized Nullspace Computations for Tall Matrices? That targets directions where $W$ is weak regardless of whether any input ever arrives there. Lifting those gains wastes the repair on directions that do not affect learning, and it can amplify noise. This is the spectral version of the null space trap LoDA describes. We include a weight-only bottom-$k$ tracker as a baseline. If it matches data-aware repair, our central hypothesis is wrong.

Why not just unfreeze everything or use bigger adapters? Plasticity loss arises precisely in networks whose weights are all trained. The LIVECell result shows full fine-tuning landing at 0.5627 AP against 0.5600 AP for the best adapter. Capacity alone does not break a representational ceiling, so extra parameters are not a substitute for fixing conditioning.

Proposed Experiments

The first experiment reproduces SingularClip's settings: Permuted MNIST, random-label CIFAR-10, Continual ImageNet with 500 binary tasks, and DMC Humanoid-Stand with SAC at high update-to-data ratio. We compare no intervention, reset, spectral-norm regularization, exact SingularClip across a $K$ sweep, the matmul-only polynomial clip, the weight-only bottom-$k$ tracker, the random-subspace and batch-subspace ablations, and our method. Tracked rank is $k \in \{16, 64, 256\}$. We report final-task accuracy, per-task steps to threshold, and RL return interquartile mean, along with per-step wall-clock and peak memory.

The second experiment is decisive. It reruns Permuted MNIST and Continual ImageNet with returning tasks, where early tasks reappear late in the stream. We measure accuracy on re-exposure before any retraining. Data-aware repair predicts higher retained accuracy than full clipping at equal plasticity. The random-subspace ablation should sit between the two.

The third experiment applies the repair to continual fine-tuning of a pretrained ViT on 20-task ImageNet-R, against Janus-LoRA's reported 77.11%. We log tracker principal angles at each task switch to test whether lag predicts where the repair helps. The fourth is the LIVECell diagnostic: restricted condition numbers of Cellpose-SAM layers, and whether restricted repair during fine-tuning moves AP off the roughly 0.56 plateau. Here a null result is informative.

The project succeeds if the returning-task experiment shows lower forgetting at matched plasticity and the random-subspace ablation loses. It fails if exact full clipping at large $K$, or the weight-only tracker, does as well.