Spending Diffusion Compute Only Where Contrastive Learning Needs It

Contrastive self-supervised learning trains an image encoder by showing it two altered copies ("views") of the same photo and teaching it that they belong together. Hand-made alterations such as crops and color changes often make this too easy, because both copies share the same background and low-level texture. Contrastive Learning with Synthetic Positives (CLSP) fixed this with a diffusion model, a generator that builds images by gradually removing noise. The diffusion model keeps the photo's object and invents a new background, producing a "hard positive" the encoder must still recognize. The fix is expensive in three ways. The images are too slow to generate during training, so they are made offline and stored on disk. They are made for every image whether or not that image needs one. And each such view adds a full encoder pass for every sample.

This note proposes making generation selective and budgeted. The encoder already computes a contrastive loss on its two ordinary views for free. We use that loss to find the images whose ordinary pair has become trivially easy. Only for those, chosen fresh each epoch, does an asynchronous worker generate a diffusion hard positive, which joins training in the following epoch. One knob, the fraction $\rho$ of samples that receive a generated view, sets both generation cost and extra encoder passes. The claim to test is that most of CLSP's accuracy gain comes from a minority of samples. If so, a small $\rho$ chosen by the free loss signal should recover most of the gain at a fraction of the total compute, generation included.

Core Claim

Two research lines treat the same cost from opposite ends, and neither accounts for the other's half. The multi-view training literature counts forward passes through the encoder but treats augmentations as free. Beyond Random Augmentations: Pretraining with Hard Views (HVP) shows that choosing hard views helps: DINO ViT-B/16 reaches 78.8% linear-probe accuracy on ImageNet-1K versus 78.2%. HVP finds those views by running all four candidates through the encoder, which slows training by 1.6x to 2.2x, and it only considers classical crops and color jitter. The synthetic-data literature counts generation cost but treats the student as a passive consumer. Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs shows that the best way to spend generation depends on the ratio of query budget to seed-set size. It also finds that expensive verification of synthetic outputs did not help the student. That result motivates us to drop heavy filtering of generated views.

We combine the two accountings into one cost per anchor image, measured relative to two-view SimCLR:

$$\frac{C}{C_{\text{SimCLR}}} = 1 + \frac{\rho\,(1+\gamma)}{2},$$

Here $\rho$ is the fraction of anchors that receive a generated view. $\gamma$ is the cost of generating one view divided by the cost of one encoder forward-and-backward pass on one view. The factor $1/2$ appears because the baseline already pays for two view passes. CLSP corresponds to $\rho=1$, with $\gamma$ paid offline and hidden from the training-time numbers.

Our hypothesis is that accuracy as a function of $\rho$ is strongly concave when anchors are ranked by "easiness." By easiness we mean a low current loss on the ordinary pair, which suggests the encoder is solving that pair through a shortcut. If the curve is concave, $\rho \approx 0.1$–$0.25$ should capture most of the $\rho=1$ gain. If accuracy grows roughly linearly with $\rho$, the claim is false.

Research Program

The view generator is CLSP's mechanism reused as-is. CLSP interpolates anchor features with noise features inside an unconditional diffusion model, using a weight of 0.1 that keeps the object and randomizes the context. We will run it with a reduced number of denoising steps and measure how $\gamma$ trades off against view quality.

The selection signal needs no extra encoder passes. Each epoch we record the per-sample InfoNCE loss of the two ordinary views, which the training step computes anyway. We rank samples by that loss and send the lowest-loss $\rho$ fraction to a generation queue running on a separate GPU. When a queued sample comes up in the next epoch, its generated view is added as a third view in CLSP's three-view objective. Generated images are kept only until they are used, so storage is bounded by one epoch's queue rather than the whole dataset. This design differs from each neighbor. CLSP is static, covers every sample, and stores everything offline. HVP spends speculative forward passes and draws from classical augmentations only. Beyond Cropping and Rotation: Automated Evolution of Powerful Task-Specific Augmentations with Generative Models (EvoAug) runs a costly evolutionary search over generative policies and is confined to few-shot regimes. Optimal Positive Generation via Latent Transformation for Contrastive Learning learns a navigator by backpropagating through a GAN. We learn nothing about the generator; we only decide where its compute goes.

Two analyses accompany the method. First, we trace the full accuracy-versus-total-GPU-hours curve as $\rho$ varies, with generation time always counted in total cost. Second, we test whether selecting easy samples is correct, rather than the hard samples HVP's logic would pick. Samples whose ordinary pair is already hard may be the ones that, like HVP's $N=8$ setting, push views toward adversarial collapse. Samples whose ordinary pair is easy are where a background swap should remove a shortcut. A secondary question is the cost of staleness: the generated view for a sample is built from that sample's loss one epoch earlier, and we will measure how much that lag hurts.

The experiments use the benchmarks where CLSP reports numbers. These are CIFAR-10 and CIFAR-100 with ResNet-18, and ImageNet-100 with ResNet-50, all evaluated by linear probe. Every run reports top-1 accuracy, total GPU-hours including generation, peak storage, and training throughput.

Baselines

The naive reference is SimCLR with its standard two views, trained under our recipe. It anchors the $\rho=0$ end of the curve. A compute-matched version is also trained for extra epochs until its total GPU-hours equal those of our chosen $\rho$. Without this version, any gain could simply reflect more compute. Beating it means higher linear accuracy at equal GPU-hours.

The strongest existing method is full CLSP with all synthetic positives pre-generated offline. CLSP reports 94.37% linear accuracy on CIFAR-10 and 79.62% top-1 on ImageNet-100, described as nearly 4 points above SimCLR. We will rerun CLSP in our codebase with the same diffusion model, epochs, and probe protocol, and charge its offline generation to its total GPU-hours. Beating it does not mean exceeding its accuracy. It means reaching at least 90% of its gain over SimCLR at no more than half its total compute and a small fraction of its storage. If CLSP gives the best accuracy at every compute level, our proposal fails.

HVP is the strongest competitor for selecting views by hardness. We will run it with $N=4$ classical candidates on the same backbones under the same epoch budget. We also report a compute-matched HVP. HVP's own reported result is 78.8% for DINO ViT-B/16 on ImageNet-1K. No reported result was found on Lacuna for HVP on ImageNet-100 or CIFAR, so those numbers will come from our runs. RSA: Reducing Semantic Shift from Aggressive Augmentations for Self-supervised Learning is a cheap aggressive-view baseline. It reports 68.1% top-1 on CIFAR-100 versus 63.9% for BYOL, with under 9% added training time. It shows how much classical aggressive augmentation achieves without any generation.

Two ablations isolate the selection rule, which is the proposal's key ingredient. Random-$\rho$ assigns generated views to a random fraction of samples at the same $\rho$ and the same generation budget. Hard-$\rho$ selects the highest-loss samples instead of the lowest. Easiness selection is justified only if it beats random-$\rho$ at matched compute. A third ablation, offline-$\rho$, generates for the same easy subset but only once, from a SimCLR checkpoint taken early in training. It measures whether online adaptation matters at all.

Why not just…?

Why not just pre-generate everything offline, as CLSP does? On CIFAR, storage is cheap, so this alternative is a real threat. The question is not feasibility but total compute and staleness. Offline generation pays $\gamma$ for every sample, including the many whose ordinary views already teach the encoder something. Its views also cannot follow the encoder as it changes, and HVP found adaptive selection mattered: a static low-IoU crop policy "fell short" of state-dependent selection. The offline-$\rho$ and full-CLSP comparisons settle this directly. If offline-$\rho$ matches online selection, the asynchronous machinery is unnecessary, and we would report the simpler offline subset method.

Why not just use HVP-style selection over classical views? HVP has to pay for forward passes before it can choose. Our selection reads a loss that training already computes. HVP's candidate pool is also limited to crops and jitter. EvoAug shows that generative views beat classical augmentation on fine-grained tasks, at 40.40% versus 30.90% on Stanford Cars. That evidence comes from few-shot transfer rather than pretraining, so our HVP baseline is still needed to show the advantage holds at equal compute.

Why not just spend the compute on longer SimCLR training or more classical views? This is the most honest null hypothesis. Tied-Augment: Controlling Representation Similarity Improves Data Augmentation reports 79.6% on ImageNet with ResNet-50 at about 30% wall-clock overhead, so cheap view-level tricks are strong. The compute-matched SimCLR and RSA runs test this. If neither CLSP nor our method beats them per GPU-hour, generated positives are not worth their cost in this regime, and we will report that.

Why not just generate cheap views for every sample with few denoising steps? That option is our curve at $\rho=1$ with small $\gamma$. Even with very cheap generation it still adds half an encoder pass per sample. The curve over $\rho$ and step count will show whether selection or cheaper sampling accounts for more of the savings.

Decisive Experiments

The central experiment traces linear accuracy against total GPU-hours for $\rho \in \{0, 0.1, 0.25, 0.5, 1\}$ under easy, random, and hard selection. It is run on CIFAR-10, CIFAR-100 and ImageNet-100, with three seeds each. The proposal succeeds if easy-$\rho$ at $\rho \le 0.25$ dominates compute-matched SimCLR, HVP, and random-$\rho$. It also requires recovering at least 90% of CLSP's measured gain at no more than half CLSP's total compute. It fails if accuracy grows linearly in $\rho$, or if random-$\rho$ matches easy-$\rho$, since either result would mean the gain is spread evenly across samples. A secondary check varies the lag between selecting a sample and training on its generated view (zero epochs using a synchronous slow run, one epoch, and five epochs) to measure the cost of staleness. A transfer check fine-tunes the ImageNet-100 encoders on Stanford Cars and Oxford-IIIT Pets, where background shortcuts should matter most. Finally, we inspect which classes and images are selected most often, to see whether "easy" really means background-solvable.