Shift-Aware Filter Decorrelation: Orthogonality Measured Under Corrupted Inputs

Convolutional networks often learn redundant filters: many kernels in a layer end up detecting nearly the same pattern. A long line of work penalizes this by pushing each layer's filters toward orthogonality, meaning their pairwise dot products are driven to zero. Separately, test-time adaptation research, which studies models that adjust themselves on unlabeled data after deployment, has found that when inputs are corrupted, the network's internal features become more redundant, and that this redundancy tracks how badly the model fails. These two findings are closely related, and nobody has joined them. Weight-space orthogonality only makes feature channels uncorrelated if the input patches are white noise. Natural images are not white, and corrupted images are even less so. A blurred image, for example, concentrates its energy in low spatial frequencies. Two filters that are orthogonal as weight vectors can produce almost identical responses on blurred input if they share low-frequency content.

We propose to regularize filters so that they stay decorrelated under the input statistics of plausibly shifted data, rather than under the identity metric that standard orthogonality assumes. The regularizer is applied during source training. At deployment the model then needs only cheap normalization-statistic updates, or an unchanged entropy-based adapter. The central question is whether this buys robustness on corruption families that were never used to compute the regularizer. If it does not, the method reduces to data augmentation with extra steps, and we treat that outcome as the main way the proposal can fail.

Core Claim

For a convolutional layer with filters stacked as rows of $W$ and input-patch covariance $\Sigma$, the covariance of the layer's outputs before normalization is $W \Sigma W^T$. Standard filter orthogonality, as in the $\|W^TW - I\|_F^2$ penalty of Xie et al. or the pairwise-cosine penalty of Paneri et al., controls this quantity only when $\Sigma = I$. Our regularizer instead penalizes the off-diagonal correlations of $W \Sigma_k W^T$ over a family of shifted input covariances $\Sigma_k$:

$$\mathcal{R}_{\text{shift}}(W) = \sum_{k} \sum_{i \neq j} h\!\left(\frac{[W\Sigma_k W^T]_{ij}}{\sqrt{[W\Sigma_k W^T]_{ii}[W\Sigma_k W^T]_{jj}}}\right).$$

Here $k$ indexes corruption operators applied to training images, $\Sigma_k$ is the covariance of the layer's input patches under corruption $k$, and $h$ is either the square or a Huber function. The Huber option caps the repulsion between filters that legitimately share structure. It follows SoC, which showed for class prompts that strict orthogonality pushes related items unnaturally far apart and harms calibration. We do not need explicit covariances. The inner term equals the channel correlation matrix of pre-normalization activations on a corrupted view of the batch, so the regularizer costs one extra forward pass per corrupted view.

We are explicit that this is an activation-decorrelation loss computed on shifted views. Its purpose is to link two literatures. With $\Sigma = I$ it recovers classical filter orthogonality. Evaluated on real test streams, it measures the redundancy that FRET removes at test time.

Relation to Prior Work

Two kinds of prior work motivate the proposal. FRET reports that feature redundancy rises monotonically with shift severity. It removes that redundancy at test time with covariance penalties (S-FRET) or an auxiliary graph network (G-FRET), treating the backbone weights as fixed. We ask whether the redundancy can be prevented upstream, so that no test-time module is needed. Perez and Yu show that restructuring filters makes networks exactly equivariant to global gain and bias shifts. They also state that this leaves blur, noise, and weather corruptions unaddressed, which are the corruptions our covariance-weighted view targets.

Other work constrains the design. PAID finds that the pairwise angular structure of pretrained weights stays stable across corruptions, and it adapts by rotating weights with Householder reflections that preserve those angles exactly. OoPk applies soft orthogonality to low-rank test-time updates, and PEGO does so for offline domain generalization with adapters. All three regularize the geometry of weights, or of updates to weights, in the identity metric. Our argument is that weight angles can stay fixed while response correlations collapse, because the input metric changes under shift. MGP protects a low-rank subspace of gradients during long streams and says nothing about filter geometry. We use its long-horizon cyclic protocol as a stress test.

Research Program

The first stage is diagnostic. For ResNets trained on CIFAR-10/100 and ImageNet, we will measure "effective redundancy" for each layer and each corruption. This is the mean squared off-diagonal channel correlation on ImageNet-C and CIFAR-C streams from Hendrycks and Dietterich. We will also measure the identity-metric redundancy of the same layer's weights. The prediction is that test-time error, both before and after adaptation, tracks the input-weighted measure and not the weight-only measure. The gap between the two should be largest for blur and noise.

The second stage trains source models with the regularizer. The corruption family used to compute the $\Sigma_k$ is always a strict subset of the evaluation corruptions. We will train on the noise-plus-blur categories and test on weather-plus-digital, and then swap the two.

The third stage deploys each source model with no adaptation, with batch-norm statistic updates as in DUA, and with Tent. Tent is run both on standard streams and under MGP's 40-round cyclic revisit protocol. The goal is to test whether decorrelated source features delay the collapse that entropy minimization is known to suffer.

Baselines

The naive reference is a standard ResNet trained with weight decay and evaluated without adaptation. The file reports a ResNet-50 mCE of 76.7 on ImageNet-C in Hendrycks and Dietterich, and 49.2% source-only error on level-5 CIFAR-10-C in DUA.

Classical identity-metric orthogonality isolates the key ingredient. We will train the $\|W^TW-I\|_F^2$ penalty of Xie et al. and the cosine penalty of Paneri et al. with the same architecture, schedule, and $\lambda$ search grid. The only difference from our method is replacing $\Sigma_k$ with $I$. We found no reported result on Lacuna for filter-orthogonality regularizers on ImageNet-C or CIFAR-C, so we will produce these numbers ourselves.

The augmentation-matched baseline trains with the identical corrupted views as extra classification data but applies no decorrelation loss. It is the most dangerous competitor for source-training robustness, because it sees exactly the same information.

Filter Normalization is the strongest filter-level robustness method in the file. It reports 67.0% versus 2.1% on ImageNet $D_S$ and 34.2% versus 28.3% on ExDark. We will run it at matched depth, and also combine it with our regularizer.

On the test-time side, the strongest competitor is PAID, which reports 42.2% mean ImageNet-C error and 11.0% on CIFAR-10-C in the continual setting. Tent reports 44.0% ImageNet-C error. G-FRET reports 36.79% ImageNet-C accuracy from a 14.70% source model. We found no reported result on Lacuna for S-FRET alone, so we will run it ourselves.

All methods share the same backbone, data order, batch sizes, and a fixed hyperparameter budget tuned on a held-out corruption not used in reporting. Our method beats the field if two things hold. First, a regularized source model with only statistic updates must match or beat Tent on held-out corruption families. Second, when paired with Tent it must lower PAID's error on those families without Householder cost. Success on the corruptions used to build $\Sigma_k$ does not count.

Why not just…?

Why not just use corruption augmentation? It exposes the network to the same shifted inputs, and it may deliver all of the gain. The augmentation-matched baseline settles this. If our method does not beat it on held-out corruption families, we conclude that the decorrelation term adds nothing and report that result. The hypothesis that the regularizer transfers rests on a specific mechanism. Decorrelation under several different metrics should push filters toward responses that are spread out in frequency, a pattern TO-RawNet observed in audio. Augmentation does not directly target that structure.

Why not just apply identity-metric orthogonality? It is cheaper and has well-established optimization benefits. The diagnostic stage tests the premise directly. If weight-space redundancy predicts shifted-feature redundancy as well as the input-weighted measure does, the covariance weighting is unnecessary, and the proposal collapses to a robustness evaluation of the Xie and Paneri penalties. That evaluation would still be missing from the literature.

Why not just decorrelate at test time with S-FRET or correlation alignment? Test-time decorrelation needs backpropagation or covariance estimates at deployment, and the file reports that it degrades under label shift. LinearTCA avoids backpropagation but aligns only final-layer features. If S-FRET on a standard source model equals our source-regularized model with statistic updates only, our method's advantage is purely deployment cost. We would report it as such, and we will measure latency and memory alongside accuracy.

Proposed Experiments

We will use CIFAR-10-C, CIFAR-100-C, and ImageNet-C at all five severities, with the category-split protocol described above. ImageNet $D_S$ and ExDark check for interaction with photometric robustness.

The metrics are top-1 error and mCE, expected calibration error, effective redundancy for each layer, and the number of rounds until accuracy collapses under the cyclic protocol.

Three results are decisive. The first is whether input-weighted redundancy, rather than weight redundancy, correlates with adaptation error across layers and corruptions. The second is whether the regularized model beats the augmentation-matched model on corruption families it never saw. The third is whether Tent on the regularized model survives more cyclic rounds than Tent on the standard model, which MGP reports falls below 1% accuracy by round 40. We will also compare square and Huber versions of $h$, with calibration error as the main readout, to test whether SoC's warning about strict orthogonality carries over to visual filters.