Holding Augmentations Just Below the Collapse Line

Self-supervised image models learn by being shown two randomly altered copies of the same photo (cropped, recolored, blurred, rotated) and being trained to give both copies the same representation. That training signal also teaches the network to throw away whatever the alterations change. Heavy random recoloring, for example, leaves models nearly colorblind, which hurts flower, bird and histology classification. The usual remedies change the augmentations once and for all, or add extra heads, branches or backbones. We propose a cheaper lever. A small controller measures, during training and without labels, whether each augmentation is currently pushing the network toward deleting the features it perturbs, and it turns that augmentation's strength up or down so the network stays just short of deleting them.

The controller is grounded in a specific theoretical result. Understanding Dimensional Collapse in Contrastive Self-supervised Learning shows that encoder weights along a feature direction vanish when the variance an augmentation injects along that direction exceeds the natural variance of the data there. That inequality can be measured directly in feature space on unlabeled batches, which gives a label-free "mastery" signal of the kind curriculum methods have lacked in self-supervised learning. The contribution is a feedback curriculum, one control loop per augmentation type, that holds this variance ratio below one. We also run a direct test of the more obvious idea, an easy-to-hard ramp of augmentation strength, which the same theory predicts should not work.

Core Claim

The collapse theory describes a property of the training dynamics near their end point. If that reading is right, an open-loop schedule that ramps augmentation from weak to strong, as Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning does for hidden-state noise in language models, should still end in collapse once the strong setting arrives. Such a schedule would change how stable optimization is, not which features survive. A closed-loop schedule behaves differently. It caps each augmentation at the strongest level that does not exceed the data variance along the directions that augmentation perturbs, so the backbone keeps color and orientation information while still receiving most of the augmentation's benefit against trivial shortcuts. We claim that this controller recovers a substantial part of the fine-grained transfer gains of LooC or Planckian Jitter with one backbone and no extra heads, while keeping the ImageNet linear-probe accuracy of standard recipes. We also claim that a plain ramp sharing the same final strength does not recover them. Either half can fail, and the experiments are designed so that each can be observed failing separately.

Mechanism

Every $N$ optimizer steps, the controller draws a small probe batch, passes it through the backbone without gradients, and computes two covariance matrices for each augmentation type $k$ (color, crop, rotation, blur). The first, $C^{k}_{\text{aug}}$, is the covariance of feature differences between two copies of the same image that differ only by augmentation $k$ at its current strength $s_k$. The second, $C_{\text{data}}$, is the covariance of features across different images. The suppression pressure on augmentation $k$ is

$$ r_k = \lambda_{\max}\!\left( (C_{\text{data}} + \epsilon I)^{-1} C^{k}_{\text{aug}} \right), $$

which is the largest ratio of augmentation-induced variance to data variance over any feature direction, with $\epsilon$ a small ridge term. This is the quantity whose crossing of one predicts collapse in the dimensional-collapse analysis. The controller updates $\log s_k \leftarrow \log s_k + \eta\,(\tau - r_k)$, where $\tau$ is a target slightly below one and $\eta$ is a step size. Strength therefore rises while there is headroom and falls when a direction is about to be erased.

The measurement is taken on backbone features, not projection-head outputs. Investigating the Benefits of Projection Head for Representation Learning and The Geometry of Projection Heads: Conditioning, Invariance, and Collapse show that the head can absorb collapse, so the head is allowed to become invariant while the controller protects the backbone underneath. The overhead is a few extra forward passes on a probe batch per $N$ steps. There are no new trainable parameters, no labels, and no bi-level optimization. Stochastic or non-invertible augmentations such as cropping, blur or CutMix are handled the same way as rotations, because the measurement needs only the ability to sample the augmentation, not to invert it.

Research Program

The work proceeds in three stages. The first is measurement only. We train standard SimCLR and VICReg (VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning) with fixed augmentations, log $r_k$ throughout training, and check whether $r_{\text{color}}$ crossing one coincides with the drop in color decodability from the backbone (a linear probe for hue statistics) and with Flowers-102 transfer. If $r_k$ does not predict suppression, the controller has no foundation, and we report that as the result.

The second stage builds the controller and compares it against open-loop schedules. The third stage takes the controller's realized strength trajectories and replays them as fixed schedules. It also replays them on a second dataset, to separate "the feedback matters" from "the controller just found good constants."

The positioning against existing curriculum work is as follows. CUDA: Curriculum of Data Augmentation for Long-tailed Recognition adapts augmentation strength per class using accuracy on augmented samples, and When to Learn What: Model-Adaptive Data Augmentation Curriculum learns a sample-level policy against labeled validation loss. Both supply the idea of feedback-paced augmentation, but both require labels, and neither looks at which features survive. EfficientCL motivates the easy-to-hard hypothesis, but it uses a fixed heuristic schedule in a different modality. Learning the Unlearned: Mitigating Feature Suppression in Contrastive Learning is the closest self-supervised curriculum. It recovers suppressed features through discrete stages that recluster the data and sample negatives from the same cluster, at a 75% parameter cost for a four-stage ViT. It leaves the augmentations themselves unchanged, whereas we change only the augmentations, continuously, at no parameter cost.

On the architectural side, What Should Not Be Contrastive in Contrastive Learning (LooC) and Amortised Invariance Learning for Contrastive Self-Supervision control invariance through extra heads or conditioned backbones. Soft Equivariance Regularization for Invariant Self-Supervised Learning is limited to invertible group actions. These papers constrain the design: they show that per-augmentation control is what matters, which is why our controller runs per augmentation rather than on a single global strength.

One paper cuts against us. Planckian Jitter finds a shape–color trade-off: once color survives, models lean on it as a shortcut and neglect shape. A controller that lowers color jitter could reproduce that trade-off, so we track ImageNet linear-probe accuracy as the shape-side cost.

Baselines

The naive reference is SimCLR, and separately VICReg, with the standard fixed augmentation recipe. Both are trained on the same data, epochs and batch size as every other arm, and evaluated from pre-projection backbone features. Evaluating before the projection head already concedes the cheapest fix to the skeptic.

The open-loop curriculum baseline ports EfficientCL's idea to vision. Each augmentation's strength ramps linearly, and in a second variant follows MADAug's tanh shape, from weak to the standard recipe's strength. Its tuning budget for the ramp length matches ours for $\tau$ and $\eta$. Beating it on fine-grained transfer while matching it on ImageNet is the core test of whether feedback, rather than gradualness, is the active ingredient.

The strongest existing methods are run at their published settings and re-run on our pipeline. Planckian Jitter reports Flowers-102 accuracy rising from 30.0% under standard color jitter to 42.7% (Planckian Jitter). LooC++ reports iNaturalist-2019 top-1 of 46.1% against 36.2% for MoCo (What Should Not Be Contrastive in Contrastive Learning). PrCL reports 88.3% on Colorful-Moving-MNIST against roughly 15% for SimCLR (Addressing Feature Suppression in Unsupervised Visual Representations). DirectCLR reports 62.7% ImageNet linear-probe accuracy at 100 epochs. These numbers come from different backbones and budgets and are not directly comparable to each other or to us. We therefore compare under matched compute and also report each method's wall-clock and parameter overhead.

Beating these methods means one of two outcomes. The controller matches Planckian Jitter's Flowers-102 gain without its dual-backbone fusion and without losing ImageNet accuracy. Or it closes most of the LooC gap on iNaturalist at single-head cost. Strict wins over LooC are not required, but a cost-matched comparison must favor us.

The key ablation is a static-matched arm. Each augmentation is fixed for the whole run at the controller's time-averaged strength, and a second variant fixes it at the controller's final strength. A further arm replays the controller's full trajectory open-loop.

Why not just…?

Why not just ramp augmentation strength from easy to hard? This is the obvious intersection of the two directions, and our core claim predicts it fails for feature preservation. The collapse condition in the dimensional-collapse paper depends on the augmentation variance in force when training ends, so a ramp to standard strength should end where standard training ends. EfficientCL's gains concern optimization stability in language-model pre-training and say nothing about suppressed features. The open-loop arm tests this directly. If the ramp matches the controller on Flowers-102 and CelebA, the feedback idea is unnecessary and we will say so.

Why not just weaken the augmentations, or use physically plausible ones like Planckian Jitter? This is the most serious threat. If the controller only learns "use less color jitter," a tuned static setting gets the same result for free. The static-matched ablation is designed to settle this. If a constant strength equal to the controller's average or final value matches it, the contribution shrinks to an automatic, label-free way of choosing that constant, which is still useful when it transfers across datasets without tuning, but is not a curriculum result. Planckian Jitter's own reported shape–color trade-off, which forced its authors into two backbones, is the evidence that a single static choice struggles to serve both kinds of task. Our ImageNet-plus-Flowers pairing measures whether the controller escapes that trade-off.

Why not just use pre-projection features or an auxiliary reconstruction branch? Pre-projection features already help: 36.0% versus 25.7% on CIFAR-100 fine-grained transfer (Investigating the Benefits of Projection Head for Representation Learning). That is why every arm, ours included, is evaluated before the head. PrCL's roughly 15% SimCLR baseline on Colorful-Moving-MNIST shows that suppression also reaches the backbone. PrCL's inpainting fixes that case, but it needs a full decoder and a static loss weight. We expect the controller not to help on Colorful-Moving-MNIST or CIFAR-MNIST, because there the suppression comes from features competing with each other rather than from augmentations. We include those benchmarks as a negative control that marks the method's scope, not as a target.

Proposed Experiments

The primary setting is ResNet-50 SimCLR and VICReg pre-trained on ImageNet-1k for 100 epochs, matching the DirectCLR comparison point, with ablations first run on CIFAR-100 using ResNet-18. Transfer is measured by linear probes on ImageNet-1k, Flowers-102, CUB-200 and iNaturalist-2019 for color and fine-grained detail, and on CelebA landmark regression for spatial and pose sensitivity, where AIL reports a 21% gain over MoCo. Detection on VOC07+12 and corruption robustness on ImageNet-C use the SSL-robustness suite. We also report the effective rank of backbone features over training, and per-attribute linear decodability of hue and rotation angle.

The decisive checks run in order. First, $r_{\text{color}}$ crossing one must precede the measured loss of color decodability; without this the mechanism fails. Second, the controller must beat both the linear ramp and the tanh ramp on Flowers-102 and CelebA at equal ImageNet accuracy; without this, feedback adds nothing. Third, the controller must beat the static-matched and trajectory-replay arms, and its replayed schedule must lose its advantage when moved to a second dataset; without this, the "curriculum" is a hyperparameter search. Fourth, it must show no gain on Colorful-Moving-MNIST, confirming the method's boundary. We will also report sensitivity to $\tau$ in the range 0.7 to 1.0. A method whose benefit depends on fine-tuning $\tau$ would repeat the pacing fragility that curriculum learning is already known for.