Spectral Side Branches with Bounded Gradients for Frozen Vision Backbones

When a large pre-trained vision model is frozen and only a small add-on is trained, the add-on can only reweight features the model already computes. On medical images, satellite tiles or depth and orientation tasks, the needed features are often missing, and lightweight tuning stalls. This failure is called representation rigidity. One cheap way to get new features out of a frozen model is to stop reading only its per-token features and also read how those features co-vary across image patches. That is a second-order statistic, the covariance matrix of patch tokens. It describes texture and part co-occurrence that no linear readout of the class token can express. Using it well requires reshaping the covariance's eigenvalues: a matrix square root or a log compresses dominant directions so they do not swamp the rest. That step means backpropagating through an eigendecomposition, and the gradient of an eigendecomposition contains terms like $1/(\lambda_i-\lambda_j)$ that blow up when eigenvalues cluster.

The low-data adaptation regime is where this blow-up is worst. With 196 patch tokens, about 1,000 training images and bf16 arithmetic, the token covariance is low-rank, and its many near-zero eigenvalues are tightly clustered. We propose a detached side branch on a frozen Vision Transformer (ViT). It projects patch tokens to a small dimension, forms their covariance, and applies a learned spectral function. The gradient is computed through divided differences, which we prove bounded by the function's slope no matter how the eigenvalues cluster. We test whether this second-order branch lifts the rigidity ceiling on the tasks where frozen features fail most, while keeping the memory savings of tuning that never backpropagates through the backbone.

Core Claim

The instability of differentiable eigendecomposition is largely an artifact of computing eigenvector gradients and then recombining them. For a spectral function $f(C) = Q f(\Lambda) Q^\top$ of a symmetric matrix $C = Q\Lambda Q^\top$, the exact gradient is

$$ \frac{\partial L}{\partial C} = Q\left(K \circ (Q^\top G\, Q)\right)Q^\top,\qquad K_{ij} = \frac{f(\lambda_i)-f(\lambda_j)}{\lambda_i-\lambda_j}\ \ (\text{and } f'(\lambda_i) \text{ when } \lambda_i=\lambda_j), $$

where $G = \partial L/\partial f(C)$ is the incoming gradient and $\circ$ is the element-wise product. This is the Daleckiĭ–Krein formula, which DeepKSPD used with L'Hôpital limits for kernel matrices. Because every $|K_{ij}| \le \sup|f'|$ and $Q$ is orthogonal, $\|\partial L/\partial C\|_F \le \sup|f'|\,\|G\|_F$. If $f$ has bounded slope on $[0,\infty)$, the backward pass cannot explode, whatever the eigenvalue clustering.

The matrix square root fails this condition, since its slope is infinite at zero. This explains why square-root pooling needs trace normalization or $\epsilon I$ shifts in rank-deficient regimes. We therefore learn $f_\theta(\lambda) = \sum_k a_k \log(1+\lambda/\tau_k)$ with $a_k \ge 0$ and $\tau_k$ bounded below. This family is monotone, has slope at most $\sum_k a_k/\tau_k$, and can represent anything from near-linear to strongly flattening shapes per task. It thereby turns the over-flattening tradeoff studied by Halley-SVD into a learned quantity rather than an iteration-design choice.

The rigidity claim is falsifiable. Gains from the branch should concentrate on tasks where frozen first-order features lag full fine-tuning the most. They should exceed a parameter-matched nonlinear head on the same patch tokens. If the second option fails, the spectral structure is not what matters.

Relation to Prior Work

The gradient side reuses and sharpens existing methods. Robust Differentiable SVD keeps an exact forward SVD and replaces the backward pass with a $K$-th order Taylor expansion. It reports that standard SVD gradients fail 100% of the time in CIFAR-100 ZCA whitening at dimension 64 and above. We target spectral functions rather than whitening, where the divided-difference form is exact rather than approximate. We compare against Taylor directly.

Halley-SVD and the Newton–Schulz and Lyapunov approach of Maji et al. avoid explicit gradients through iterations. All three train covariance pooling end to end inside a fully trained network, with full-rank covariances from convolutional feature maps. None operates on a frozen transformer under data scarcity. Dobi-SVD stabilizes SVD gradients only to learn compression ranks offline.

On the adaptation side, the evidence for rigidity motivates the work. The pathology-CLIP study (Bridging the Pathology Domain Gap) reports that lightweight adapters fail when base features are missing. Shakeri et al. find that prompt methods often do not beat a linear probe when domain coverage is lacking.

The spectral PEFT methods Spectral Adapter and ABBA decompose weights once, offline, and never differentiate through the decomposition. Our branch decomposes activation statistics on every forward pass. It adds a nonlinearity those weight-space methods do not have.

The closest architectural precedent is Res-Tuning. Its bypass variant detaches the tuner from the backbone and cuts training memory by up to 70%. Its tuners are first-order, so it inherits the backbone's rigidity. We keep its detached design and change what the branch computes.

We considered and rejected a more obvious combination: re-decomposing $W_0+\Delta W$ inside a LoRA-style adapter with differentiable SVD. If the loss depends on the weight only through the reconstructed matrix, the gradient through the SVD is the gradient to the weight itself. The construction adds instability and no expressivity.

Research Program

The branch reads patch tokens from a few frozen ViT-B/16 layers. A trainable linear map sends each layer's tokens to dimension $d$ (64 to 256). The branch forms the centered token covariance per image, applies $f_\theta$ via a forward eigendecomposition in fp32 (cheap at these sizes), and vectorizes the upper triangle. It fuses the result with the class token in a linear classifier.

Gradients flow to the projections and to $\theta$ through the divided-difference backward pass. Near-equal eigenvalue pairs, those with $|\lambda_i-\lambda_j|$ below a tolerance, use $f'$ at their midpoint. Nothing backpropagates into the backbone.

We will build this as a drop-in head in the SSF codebase, which already contains VTAB-1k and fine-grained training scripts for ViT-B/16. Three analyses accompany the method:

  1. Stability study. We sweep $d$, precision and number of tokens. We record failure rate, maximum gradient norm and the fraction of near-degenerate eigenvalue pairs, comparing the bound above against observed norms.
  2. Learned spectral shapes. We characterize the learned $f_\theta$ per task, asking whether specialized and structured tasks learn more flattening than natural ones.
  3. Rigidity diagnosis. For each VTAB task we regress the branch's gain on the gap between a frozen linear probe and full fine-tuning.

Baselines

The naive reference is a linear probe on the frozen class token, run with the same augmentation and tuning budget as everything else. It defines how rigid the backbone is per task. The file contains no reported VTAB-1k linear-probe number; we will measure it. Full fine-tuning is reported at 65.57% top-1 on VTAB-1k and VPT-Deep at 69.43% by SSF; both serve as further reference points.

SSF itself is the strongest coupled competitor. It scales and shifts features throughout the backbone and reaches 73.10% VTAB-1k average and 90.72% FGVC average with 0.24M parameters. We rerun it from the official code on our splits.

Res-Tuning is the strongest overall and the most directly comparable, since it is also a detached branch. It reports a 74.10% VTAB-1k average (Res-Tuning), though that figure may come from its coupled rather than bypass variant. We report both variants under identical parameter counts, epochs and hyperparameter search budgets.

Beating Res-Tuning means one of two outcomes:

  • higher VTAB-1k average than the bypass variant at equal or lower training memory, with gains concentrated in Specialized and Structured groups; or
  • matching the coupled Res-Tuning average while never backpropagating through the backbone.

We also test SSF combined with our branch, because the two may be complementary.

The pooling baselines share our exact branch and projections and swap only the spectral step:

  • iSQRT-COV (Newton–Schulz square root);
  • Halley-SVD, which surpasses iSQRT-COV by 1.2% on average across CUB, Dogs and Cars in its paper;
  • Taylor-gradient SVD with a square root;
  • plain autograd eigendecomposition with an $\epsilon I$ shift and gradient-norm clipping.

Two ablations isolate the key ingredients. A fixed $f(\lambda)=\sqrt{\lambda+\epsilon}$ with our backward pass tests whether learning $f$ matters. Autograd versus divided differences under the same learned $f$ tests whether the gradient formulation matters.

Why not just…?

Why not just use SSF or another linear modulation? Linear modulation reweights existing features, which is exactly the rigidity it cannot escape. Even on ImageNet-1K, SSF falls slightly below full fine-tuning (83.10% vs. 83.58% in SSF). Still, SSF beats full fine-tuning on VTAB-1k, so it is a serious threat. The per-task rigidity regression settles the question. If our gains do not concentrate on tasks with large probe-to-full-tuning gaps, the rigidity story is wrong.

Why not just use Newton–Schulz square-root pooling, which needs no eigendecomposition gradient? It is stable, but Halley-SVD shows it over-flattens dominant eigenvalues and loses discriminative signal. It also fixes the spectral shape to a square root, while our function is learned. This is the most likely baseline to tie us, especially on natural-image tasks. If iSQRT matches the learned branch, we will report that the second-order statistic, not the gradient machinery, carries the benefit.

Why not just backpropagate eigendecomposition with an epsilon shift and gradient clipping? Robust Differentiable SVD reports total failure of standard SVD gradients at dimension 64 and above. Clipping also rescales the gradient's direction across the whole update. The stability sweep measures this directly in our rank-deficient, bf16 regime.

Why not just put a parameter-matched attention or MLP head on the patch tokens? This is the alternative most capable of explaining away our result. Any gain might come simply from reading patch tokens nonlinearly. We include it at equal parameters and memory. Our claim survives only if the spectral branch wins on Specialized and Structured VTAB tasks and on pathology.

Proposed Experiments

The main experiments use VTAB-1k on ViT-B/16 across all 19 tasks, grouped Natural, Specialized and Structured. They also include the five fine-grained datasets used by SSF, among them CUB-200-2011, Stanford Cars, Stanford Dogs and Oxford Flowers.

For a strong-rigidity setting we add PCam under label scarcity, matching the pathology-CLIP protocol of 0.1% labels. That paper reports only a relative 19% gain over CLIP adapters, so we rerun its adapter baseline rather than quote an absolute number.

Metrics are top-1 accuracy per task and group average, plus peak training memory and throughput. For stability we track failure rate, gradient-norm statistics and the near-degenerate eigenvalue fraction across $d \in \{32, 64, 128, 256\}$ in fp32 and bf16.

Three checks decide the outcome:

  1. Gradient bound. The divided-difference branch must train with zero failures in bf16 at $d=256$, where autograd with $\epsilon I$ is expected to fail. Measured gradient norms should stay under $\sup|f_\theta'|\,\|G\|_F$.
  2. Rigidity concentration. Gain over Res-Tuning-Bypass must correlate positively with the per-task probe-to-full-tuning gap.
  3. Spectral structure matters. The branch must beat both the parameter-matched attention head and iSQRT-COV on Specialized and Structured VTAB and on PCam.

If the first check passes but the third fails, the contribution narrows to a stable backward pass for spectral functions in low-rank regimes. That is still useful, but it would not lift the rigidity ceiling.