Truncated Newton–Schulz Whitening for Streaming Synthetic-to-Real Adaptation

A network trained on rendered images sees real photographs whose features are shifted and differently correlated. The cheapest known fix changes no weights at all. At test time it re-estimates each layer's feature statistics on the incoming real data and maps them back onto the statistics the network saw during training. Today this is almost always done per channel, as in batch-norm statistics adaptation, which corrects means and variances and ignores correlations between channels. Correcting the full covariance requires a matrix inverse square root, $\Sigma^{-1/2}$, of a covariance estimated from a single small test batch. That matrix is expensive to compute exactly, and it is badly estimated because the batch has fewer samples than the feature has dimensions. We propose computing it with a few steps of Newton–Schulz iteration, a method that approximates an inverse square root using only matrix multiplications, warm-started across the test stream.

The central observation is that stopping the iteration early is not only a speed trick. When started from a scaled identity, Newton–Schulz converges fastest along high-variance directions of the covariance and slowest along low-variance ones. The low-variance directions are exactly the ones a small batch estimates worst. A truncated iteration therefore acts as a built-in spectral shrinkage: it whitens the reliable directions and leaves the noisy ones nearly untouched, and it does so in pure matrix-multiplication kernels that run at full speed on GPUs and NPUs. The question the proposal answers is whether this cheap, self-regularizing full-covariance correction beats per-channel statistics adaptation on synthetic-to-real benchmarks without the heavy augmentation that makes current continual adaptation methods slow.

Core Claim

Full-covariance test-time recoloring is useful for synthetic-to-real shift only if the whitening step is regularized. Recoloring here means whitening real-stream features with the target covariance and then re-imposing the stored source covariance. A small, fixed number of warm-started Newton–Schulz iterations supplies that regularization implicitly, at a cost of a few batched matrix multiplications per layer. We expect it to match or exceed exact eigendecomposition with a tuned ridge penalty, at a fraction of the latency.

To make the shrinkage claim precise, consider the Newton iteration for the inverse square root of a covariance $A=\Sigma/c$ that has been scaled by a constant $c$ so its eigenvalues lie in $(0,1]$:

$$X_{k+1} = \tfrac{1}{2} X_k \left(3I - X_k^2 A\right), \qquad X_0 = I.$$

Along an eigendirection with eigenvalue $\lambda$, the scalar iterate grows by at most a factor of $3/2$ per step while $x_k^2\lambda \ll 1$. After $K$ steps the gain applied to that direction is therefore roughly $\min(\lambda^{-1/2}, 1.5^K)$. $K$ is thus a hard cap on noise amplification, set by an integer rather than a tuned continuous penalty. This matters because, as in A General Algorithm to Calculate the Inverse Principal p-th Root of Symmetric Positive Definite Matrices, which generalizes this family of iterations to arbitrary inverse $p$-th roots and studies the cost of different expansion orders, the numerical-analysis goal is full convergence. We deliberately do not converge. The iteration uses products like $X_k^2 A$ that assume near-commuting iterates, so we use the numerically stable coupled form for the actual implementation.

On a stream, restarting from the identity every batch wastes work, but warm-starting from the previous estimate lets the cap grow without bound. We therefore warm-start from a blend, $X_0 = (1-\beta)X_{\text{prev}} + \beta I$, with $\beta\in[0,1]$. This carries information across batches while pulling poorly supported directions back toward identity, and the blend doubles as a forgetting mechanism when the domain shifts.

The adapted feature is $\hat h = \Sigma_s^{1/2} X (h-\mu_t) + \mu_s$. Here $h$ is a real-stream feature, $\mu_t$ is the running target mean, $\mu_s$ is the stored source mean, and $\Sigma_s^{1/2}$ is the source square-root covariance, computed exactly once offline from synthetic data. This whitening-and-recoloring map moves the target Gaussian onto the source Gaussian. It is not the Bures–Wasserstein optimal map; comparing the two is part of the program.

Relation to Prior Work

Three papers motivate the gap. Deep CORAL: Correlation Alignment for Deep Domain Adaptation showed that aligning source and target covariances helps adaptation, but it does so as a training loss that needs source data and backpropagation. We perform the alignment as a closed-form feature transform at test time. Efficient Multi-Domain Learning by Covariance Normalization whitens and recolors features with tiny learned adapters, using exact offline PCA per task. We need neither labels nor offline eigendecompositions. Affine Transport for Sim-to-Real Domain Adaptation uses the closed-form Gaussian transport map for simulated-to-real robot dynamics, cutting hockey-puck prediction error from 0.73 to 0.08. Its authors note that exact matrix roots become costly and ill-conditioned in high dimensions, which is precisely the regime of deep features.

Two papers supply the method. Stochastic Whitening Batch Normalization is the closest in spirit. It replaces exact and Newton-based whitening (IterNorm) with first-order stochastic updates of a whitening matrix, and it reports that imperfect whitening regularizes. It trains on static data and was never tested under shift or at test time. We keep its insight that partial whitening helps, but obtain the partialness from the truncation structure of a second-order iteration rather than from slow gradient steps. When Good Enough Is Optimal: Multiplication-Only Matrix Inversion Approximation for Quantized Gated DeltaNet shows that replacing sequential substitution with a truncated series yields up to 5× kernel speedups on NPUs. It motivates our latency claim and the target hardware.

Two papers constrain the design. Continual Test-Time Domain Adaptation and Hybrid-TTA: Continual Test-time Adaptation via Dynamic Domain Shift Detection show that pseudo-label and augmentation-based adaptation reaches strong accuracy but at low throughput and with risk of error accumulation. A statistics-only method must at least beat per-channel statistics adaptation to be worth its cost. We also stress-test it on repeated domain cycles, because running-statistic methods can drift.

Research Program

We will build a drop-in module that replaces each normalization layer's statistics at test time. To keep matrices small and batched, whitening is applied group-wise over channel groups (for example 32–128 channels per group), following IterNorm-style grouping. Each layer holds a running target mean and covariance, a stored source mean and square-root covariance, and the warm-started Newton–Schulz iterate. The free choices are the number of iterations $K$, the blend $\beta$, the group size, and which layers are adapted.

The analytical part characterizes the effective spectral filter that $(K,\beta)$ induce on the running covariance. It then compares that filter with the ridge and Ledoit–Wolf style shrinkage that exact eigendecomposition would need for small batches. The empirical part asks three questions. Does full-covariance correction beat per-channel correction on synthetic-to-real shift at all? Is truncation-as-shrinkage competitive with exact whitening plus tuned shrinkage? Does the method survive continual, cycling shift where running statistics are known to be fragile?

A secondary study replaces whitening-and-recoloring with the true Gaussian transport map. That map is itself computable by coupled Newton–Schulz square-root iterations, which extends Affine Transport for Sim-to-Real Domain Adaptation to feature dimension. The study tests whether optimality of the transport matters once estimation noise dominates.

Baselines

The naive reference is the unadapted source model, together with per-channel batch-norm statistics adaptation, which re-estimates only means and variances on the test batch. Per-channel adaptation is the real reference point because our method is its strict full-covariance generalization. Continual Test-Time Domain Adaptation reports 20.4% error on CIFAR-10-C for it. Beating it by a clear margin at comparable latency is the minimum bar.

The strongest accuracy competitors are the weight-updating methods. On Cityscapes-to-ACDC, Hybrid-TTA reports 62.2 mean mIoU at about 2 frames per second, and CoTTA reports 58.6 mIoU over ten cycles. On CIFAR-10-C, CoTTA reports 16.2% error. On VisDA-C, AdaContrast (Contrastive Test-Time Adaptation) reports 86.8% target accuracy, against 83.0% for SHOT. We will run all of them on the same backbones, batch sizes, stream orders and GPU, with published hyperparameters plus an equal small tuning budget on a held-out corruption or domain. For VisDA semantic segmentation (synthetic driving scenes to real) we found no reported result on Lacuna for any test-time method, so we will establish numbers for all methods ourselves. We do not expect a statistics-only method to beat AdaContrast or Hybrid-TTA outright. Success on this axis means closing most of the gap from per-channel adaptation to these methods at several times their throughput. A second route to success is being additive when combined with them, for instance used as the normalization inside TENT.

Exact whitening is the most important baseline scientifically. It computes each group's $\Sigma^{-1/2}$ by batched eigendecomposition, with ridge or Ledoit–Wolf shrinkage and the shrinkage strength tuned on the same held-out split. The first-order alternative is an SWBN-style stochastic whitening update run online with a tuned learning rate. The key ablations isolate the proposal's ingredient: Newton–Schulz run to convergence (no implicit shrinkage), cold-started versus warm-started iterations, $\beta=0$ versus tuned $\beta$, and diagonal-only versus full group covariance.

Why not just…?

Why not just use exact eigendecomposition with a ridge penalty? This is the real threat. If exact whitening with tuned shrinkage matches truncated Newton–Schulz in accuracy, the shrinkage claim fails and the contribution reduces to latency. The latency advantage would still be meaningful, because batched small eigendecompositions run sequential kernels, while When Good Enough Is Optimal: Multiplication-Only Matrix Inversion Approximation for Quantized Gated DeltaNet reports large speedups from multiplication-only replacements. We will report both accuracy at matched shrinkage-tuning budget and wall-clock time per batch. We will also check whether the truncation filter's shape (a gain cap rather than an additive ridge) matters in the small-batch regime, where the covariance is rank-deficient and ridge must be large.

Why not just adapt per-channel statistics, which already works? Per-channel adaptation cannot undo changes in how channels co-vary. Synthetic-to-real shift in texture and lighting plausibly rotates feature correlations, not only their scales, and the covariance-alignment gains in Deep CORAL: Correlation Alignment for Deep Domain Adaptation are indirect evidence of this. The diagonal-versus-full ablation settles it directly. If full covariance adds nothing on VisDA, the proposal's premise is wrong, and we will say so.

Why not just update a whitening matrix with stochastic gradients, as in SWBN? Stochastic Whitening Batch Normalization converges linearly and was tuned over long training runs on fixed data. Under a stream that switches domains every few hundred batches, such as Cityscapes-to-ACDC cycling, it may lag behind each shift. Our online SWBN baseline, with a tuned learning rate, measures how many batches each method needs to recover after a switch.

Why not just run pseudo-label or augmentation methods like CoTTA? They are more accurate today but slow. Hybrid-TTA's roughly 2 frames per second is reported as a 20-fold improvement over augmentation-heavy methods. They also risk error accumulation from bad pseudo-labels. Our method uses no labels and no gradients, so the comparison is an accuracy–throughput frontier plus a stacking experiment, not a head-to-head accuracy race.

Proposed Experiments

The primary benchmark is VisDA-C classification (VisDA: A Synthetic-to-Real Benchmark for Visual Domain Adaptation), with a ResNet source model trained on renders and adapted online on real images. We sweep batch sizes from 16 to 128 because the shrinkage claim predicts the largest advantage at small batches. VisDA segmentation serves as the synthetic-to-real dense task. CIFAR-10-C and Cityscapes-to-ACDC, which are real-to-corrupted rather than synthetic-to-real, test continual and cyclic stability, run for ten cycles. On the metric side we report accuracy or mIoU, calibration error following Contrastive Test-Time Adaptation, and per-batch latency on GPU and, where available, an NPU.

The decisive checks are few. First, on VisDA-C with small batches, a curve of accuracy against $K$ should rise and then fall. The fall is the implicit shrinkage we claim; a monotone rise toward the converged solution would refute the claim. Second, the measured gain spectrum of the truncated iterate should track $\min(\lambda^{-1/2},1.5^K)$, and its accuracy should be matched only by exact whitening with shrinkage tuned per batch size. Third, full covariance must beat diagonal adaptation on VisDA by more than seed variance. Fourth, over ten Cityscapes-to-ACDC cycles, mIoU with $\beta>0$ should not decay, while $\beta=0$ is expected to drift. Finally, a secondary run on the Hopper, Ant and HalfCheetah dynamics tasks from Affine Transport for Sim-to-Real Domain Adaptation will check that the iterative map reproduces exact affine transport in low dimension before we trust it in high dimension.