Band-Wise Variance Floors for Frequency-Filtered Self-Supervised Encoders

Self-supervised joint-embedding methods train an image encoder by showing it two randomly distorted copies of the same picture and asking for matching outputs. VICReg is the clearest example. It adds two statistical penalties so the encoder cannot cheat by giving every image the same output: a floor on how much each output dimension varies across a batch, and a penalty on correlations between dimensions. Those penalties inspect only the output vector. They say nothing about which parts of the image the variation comes from. We argue this blind spot matters. The standard distortions (blur, crop-and-resize) mostly destroy fine, high-frequency detail, so an invariance loss rewards the encoder for ignoring that detail. The variance and covariance terms can then be satisfied entirely with coarse, low-frequency content. The resulting collapse is hidden: the embedding looks full-rank, yet its information comes from a narrow slice of the image spectrum.

The problem gets sharper once you put a learnable frequency filter (a layer that transforms features with a Fourier transform, reweights frequencies with trainable or input-dependent gates, and transforms back) inside such an encoder. Under an invariance loss, the easiest move for that filter is to become a low-pass filter that deletes whatever the augmentations perturb. This is a new route to collapse, and VICReg's output-level terms cannot see it. We propose to measure this "spectral collapse" directly. The fix is to split the encoder's features into frequency bands, using content-adaptive band boundaries, and to give each band its own variance floor and a cross-band decorrelation penalty. The encoder then cannot silence any band. We test whether this improves linear-probe accuracy, robustness to image corruptions, and resistance to frequency shortcuts.

Core Claim

The claim has two parts. The first is diagnostic. In VICReg-trained encoders, the sensitivity of the embedding to removing a given frequency band is heavily concentrated in low frequencies. This concentration is driven by the augmentation pipeline. It gets worse when a learnable spectral filter is placed in the encoder: the filter's learned gain on high and mid bands drifts toward zero during training.

The second part is a remedy. A per-band variance floor, combined with a penalty on covariance between bands, prevents this drift and yields representations that use a wider part of the spectrum. The remedy should help most where spectral content carries semantic information that augmentations corrupt, and on corrupted or out-of-distribution test data.

Relation to Prior Work

Four groups of work set up this proposal.

The first group supplies the collapse framing and explains why output-level statistics are not enough. Understanding Dimensional Collapse in Contrastive Self-supervised Learning shows that dimensional collapse arises when augmentation variance along a direction exceeds data variance there. It studies this for crops and color jitter, not frequency structure. Our hypothesis is the frequency-domain version of their mechanism: blur and resize inject large augmentation variance into high-frequency directions. On Feature Decorrelation in Self-Supervised Learning shows that per-dimension standardization stops complete collapse but not dimensional collapse, and that full whitening helps. We make the analogous move one level up: a floor on overall variance does not stop collapse across bands. VICReg supplies the objective we build on. It treats the embedding as one undifferentiated vector.

The second group motivates the spectral side and also constrains it. Frequency Shortcut Learning in Neural Networks finds that supervised CIFAR-10 models rely on fewer than 20% of Fourier frequencies, often texture-like high frequencies, and fail badly out of distribution. FUSE reports the opposite tendency, a low-frequency bias, in supervised re-identification. This contradiction bears directly on us. A band-wise variance floor could push the encoder into a high-frequency shortcut rather than toward useful detail. So shortcut analysis has to be a primary evaluation, not an afterthought. FUSE also supplies our method component: learnable radial masks that split a 2D Fourier spectrum into low, mid and high bands. FUSE trains these with supervised losses and a spectral energy alignment term. It applies no self-supervised objective and no covariance penalty.

The third group is generative spectral pre-training. Masked Frequency Modeling avoids collapse by reconstructing randomly masked frequency bands. Learnable FFT-VAE predicts per-input filter parameters inside a VAE and notes that it needs an auxiliary regularizer to keep the filter shapes from degenerating. That observation is exactly the failure we expect to be worse in joint-embedding training, where there is no reconstruction target to anchor the filter.

The fourth group is the closest existing link between frequency content and VICReg. Sequential Multi-Dimensional Self-Supervised Learning for Clinical Time Series applies VICReg to high-sample-rate ECG embeddings, but it does no spectral decomposition. Intra-Instance VICReg uses "spectral" in the graph-kernel sense and involves no Fourier transforms.

To our knowledge from this literature, no one has applied variance or covariance constraints per frequency band. No one has studied whether a learnable spectral filter collapses under a joint-embedding objective.

Method

We insert a band-splitting module after an early stage of a ResNet. It computes the 2D Fourier transform of each feature map and multiplies it by three soft radial masks: low, mid and high. Each band's features are then transformed back and passed through the remaining layers in a separate channel group. The result is a representation $h = [h_L, h_M, h_H]$. The mask boundaries are predicted per image by a small network that reads pooled spectral energy, in the style of Learnable FFT-VAE. A static-boundary version is kept as an ablation.

Standard VICReg is applied to the concatenated projector output. Two extra terms are added on the pre-projector band groups:

$$\mathcal{L} = \mathcal{L}_{\text{VICReg}}(Z,Z') + \mu_b \sum_{b\in\{L,M,H\}} v(h_b) + \nu_b \sum_{b\neq b'} \frac{\|C(h_b,h_{b'})\|_F^2}{d}.$$

Here $v(\cdot)$ is VICReg's hinge on per-dimension standard deviation, applied to band $b$'s features over the batch. $C(h_b,h_{b'})$ is the cross-covariance between two bands' features. $d$ is the per-band dimension. $\mu_b$ and $\nu_b$ are weights. The variance floor stops any band from being silenced. The cross-band term stops the encoder from satisfying the floor by copying low-frequency content into the high-band channels.

For diagnosis we define band sensitivity. $S_b$ is the expected squared change in the embedding when band $b$ is removed from the input image, normalized by the embedding's total variance. We report $S_b$ on a grid of radial bands for every trained model. We also track the learned filter gain per band during training.

Research Program

The program runs in three stages. First, we measure $S_b$ for plain VICReg on CIFAR-10, STL-10 and ImageNet-1K at 100 epochs, and again with Gaussian blur and aggressive resizing removed. This establishes whether spectral concentration exists and whether augmentations cause it. Second, we add the learnable filter without band-wise terms and record whether its high-band gain decays. That result would directly show the new collapse pathway. Third, we add the band-wise terms and measure accuracy, robustness and shortcut behavior. If stage one shows no concentration beyond what the data spectrum itself implies, the premise is wrong and we will report that. We expect that outcome would still be informative about why augmentations matter.

Baselines

Plain VICReg. The naive reference is VICReg with the standard ResNet-50 and 8192-d expander, run with the exact recipe from VICReg. It is the strongest competitor because it is the base objective. Any gain must come from the band-wise terms, not from a different training setup. At 100 epochs on ImageNet-1K, On the Duality Between Contrastive and Non-Contrastive Self-Supervised Learning reports roughly 68% top-1 linear-probe accuracy for VICReg. VICReg reports 73.2% for the full recipe. We rerun it ourselves with identical data, epochs and augmentations. We give VICReg the same number of hyperparameter trials over $\lambda,\mu,\nu$ as our method gets over its full weight set, and match parameter count by widening VICReg's backbone channels to equal the band module's overhead.

Shuffled-DBN. The strongest direct-decorrelation competitor is Shuffled-DBN from On Feature Decorrelation in Self-Supervised Learning. It reports 89.50% on CIFAR-10 and 65.18% on ImageNet. It tests whether stronger whitening of the whole vector already achieves what band-wise constraints do.

Ablations. Three ablations isolate the key ingredient. The first is the filter-equipped encoder trained with plain VICReg: the architecture is identical, only the band-wise terms are dropped. The second uses static learned band boundaries instead of per-image ones. The third keeps the variance floors but drops the cross-band covariance term.

What counts as winning. We require three things. Linear-probe accuracy must be at or above rerun VICReg, beyond seed variance over three seeds. ImageNet-C mean corruption error must be lower; we found no reported VICReg ImageNet-C result on Lacuna, so we measure it ourselves. The spectrum of $S_b$ must be flatter than the rerun VICReg's, while the shortcut test below shows no increase in shortcut reliance.

Why not just…?

Why not just remove blur and resize from the augmentations? If augmentations alone cause spectral concentration, removing them might restore high-frequency use for free. This is a real threat, and stage one tests it directly by measuring $S_b$ and accuracy with and without these augmentations. The concern is that these augmentations are among those the joint-embedding literature relies on, so removing them may cost accuracy. Our method keeps them and constrains their spectral side effect. If augmentation removal matches our method on both accuracy and $S_b$, the band-wise terms are unnecessary, and we will say so.

Why not just use fixed band splits? Static radial bands with band-wise variance floors could do everything the adaptive version does. FUSE and Learnable FFT-VAE both report advantages for learnable over fixed filtering, but only in supervised or generative settings. Our static-boundary ablation settles the question for joint embeddings. We treat per-image boundaries as a hypothesis to test, not as the contribution. The core contribution is the band-level variance constraint and the collapse diagnosis.

Why not just whiten harder, or use generative spectral reconstruction? Whitening, as in Shuffled-DBN, decorrelates dimensions but cannot tell which input frequencies those dimensions draw on. Running it as a baseline and measuring its $S_b$ shows whether it incidentally fixes spectral concentration. Masked Frequency Modeling avoids collapse through reconstruction. However, it asks a different question, about generative pre-training with fine-tuning, and it uses random rather than learned masks. We include it as a spectral reference point on ImageNet-C rather than as a direct competitor on linear probing.

Proposed Experiments

Development runs use CIFAR-10, CIFAR-100 and STL-10 with ResNet-18. Headline runs use ImageNet-1K with ResNet-50 at 100 epochs, evaluated by linear probe and on ImageNet-C.

Shortcut test. We use the protocol from Frequency Shortcut Learning in Neural Networks. We compute dominant frequency maps for linear probes on CIFAR-10 features and evaluate transfer to ImageNet-10, including the setting with dominant frequencies removed.

Decisive checks. Three results decide the case.

The first concerns filter drift. In the filter-plus-plain-VICReg ablation, the learned high-band gain must fall during training; if it does not, the proposed collapse pathway does not exist. It must stay bounded once band-wise floors are added.

The second concerns the robustness gain. Our method must beat VICReg on ImageNet-C by more than the augmentation-removal baseline does, at no loss in clean linear-probe accuracy.

The third concerns shortcuts. Dominant frequency maps must not shrink toward a narrow high-frequency set, and ImageNet-10 transfer must not degrade. If the high-band floor improves in-distribution accuracy while shrinking dominant frequency maps, we have traded collapse for a shortcut, and the proposal fails on its own terms.

A secondary probe revisits the expander question. We rerun with a 2048-d projector, because the drop from a large to a small projector reported by On the Duality Between Contrastive and Non-Contrastive Self-Supervised Learning (3.5 to 4 points) may partly reflect spectral redundancy that band-wise constraints remove.