Spending the Image Budget Where the Heads Disagree

Classifiers often learn shortcuts. A bird classifier may learn "water background means waterbird" in place of the bird's shape. One family of fixes trains several prediction heads that must agree on the labeled training data but disagree on unlabeled images where the shortcut and the true feature point to different answers. Once the first head has taken the shortcut, the others are pushed toward other features. These methods need images that break the shortcut. When the training set contains none, as in complete-correlation benchmarks where every waterbird sits on water, the only source is a generative model, such as a diffusion model (a generator that turns random noise into images by gradually denoising it). Current pipelines generate blindly. They either render a large balanced synthetic dataset over subgroups that must be named in advance, or sample passively from a partially trained generator. Rendering images at scale is expensive, and most of the images rendered this way teach the classifier nothing.

We propose to make generation respond to the classifier. The disagreement among the heads acts as a live, label-free signal that decides which candidate edits are worth fully rendering. Our core claim is that, under a fixed generation budget, spending renders where the current heads disagree most matches the worst-group accuracy of blind generation with roughly an order of magnitude fewer rendered images. A second claim is less obvious. The same disagreement objective can be satisfied by exploiting the generator's own visual artifacts rather than the true feature. We handle this with a cheap control: agreement anchors that carry the generator's artifacts but not its counterfactual edits. This replaces the costly synthetic-pretraining stage that earlier work needed to defeat the artifact shortcut.

What already exists and what is missing

The disagreement line supplies the learning objective. Agree to Disagree (D-BAT) trains models one after another, each disagreeing with its predecessors on unlabeled out-of-distribution data. Diversify and Disambiguate (DivDis) trains many heads on a shared backbone, minimizing mutual information between them on target data, and then uses a few labels to pick a head. It reports 82.4% worst-group accuracy on Waterbirds-CC. Both assume a suitable unlabeled pool exists, and both leave open what to do when it does not. That gap motivates this proposal.

Three papers bring in generation, and each leaves a specific hole. DiffDiv trains a diffusion model from scratch on the biased data. It samples from a hand-picked "originative" intermediate checkpoint and applies ensemble disagreement to those samples. The ensemble never tells the generator what to make, and the authors note that finding the checkpoint is manual. From Fake to Real (FFR) makes the central negative point. When real and synthetic images are mixed, a model can learn the compound shortcut formed by the spurious attribute together with the real-or-synthetic source. FFR avoids this by pretraining entirely on a large balanced Stable Diffusion corpus, which requires knowing the spurious attribute and paying for full-scale generation. That result constrains our design: any mixed-data scheme must show it is not learning the source artifact. DisGUIDE supplies the mechanism. It uses disagreement between two clone models to steer a generator and cuts victim queries by up to 64.95% in model extraction. However, it works on balanced CIFAR data with a GAN (a generator trained against a discriminator), and it never touches shortcut learning. DECORATE is the old ancestor: it diversifies ensembles with artificial points that carry oppositional labels, but its generators are only simple sampling schemes.

Our proposal differs from all four. The generator is off the shelf and never retrained. Disagreement is used as an acquisition score over cheap drafts, not as a training loss for the generator. The artifact problem is handled inside a single training stage. Generation cost is a measured axis, not an afterthought.

Core Claim

Let $h_1,\dots,h_N$ be heads on a shared backbone. Training minimizes

$$ \mathcal{L} = \mathrm{CE}(D_{\text{real}}) + \lambda_a \,\mathrm{Agree}(D_{\text{anchor}}) - \lambda_d \,\mathrm{Dis}(D_{\text{edit}}). $$

The terms are as follows. $\mathrm{CE}$ is the classification loss averaged over heads on the labeled real data. $\mathrm{Dis}$ is DivDis's pairwise mutual-information disagreement between heads. $\mathrm{Agree}$ is the negative of that quantity, which penalizes disagreement. $D_{\text{edit}}$ holds the generator's counterfactual edits of training images. $D_{\text{anchor}}$ holds the generator's null edits, where a training image is passed through the same image-to-image pipeline with an empty edit instruction. Anchors carry the generator's artifacts but keep the original correlation intact. A head that flips its predictions whenever it detects synthetic texture therefore pays on the anchors as much as it gains on the edits. The only way to disagree cheaply is to respond to content that the edit actually changed.

Rendering is allocated by acquisition. Each candidate edit $x$ is first produced as a low-step draft $\tilde{x}$, costing a small fraction of a full render. It is then scored by

$$ a(\tilde{x}) = \mathrm{H}\Big(\tfrac{1}{N}\textstyle\sum_i p_i(\tilde{x})\Big) - \tfrac{1}{N}\textstyle\sum_i \mathrm{H}\big(p_i(\tilde{x})\big), $$

where $p_i$ is head $i$'s predicted class distribution and $\mathrm{H}$ is entropy. This is the standard mutual-information measure of committee disagreement. Only the top-scoring drafts are fully rendered and added to $D_{\text{edit}}$. Candidate edits come from an unsupervised source: a captioner describes each training image, and an edit is produced by swapping one caption phrase (background, object, texture, or pose) for an alternative phrase drawn from the dataset's own captions. The spurious attribute never needs to be named. Disagreement also feeds a simple bandit over edit families, so the budget moves toward the phrase types that keep splitting the heads.

Research Program

We will build the loop on a pretrained ResNet-50 backbone with DivDis-style heads and an off-the-shelf image-to-image diffusion editor. The loop alternates between a few epochs of training under $\mathcal{L}$ and a generation round with a fixed render budget per round. Every experiment logs total generator GPU-hours, split into drafting, rendering and captioning, alongside accuracy. The main object of study is the curve of worst-group accuracy against generation compute, not a single final number.

The analysis has three parts. First, we examine what the acquisition score selects. We check whether high-disagreement drafts concentrate on shortcut-breaking edits, such as background swaps on Waterbirds, and whether this concentration fades once one head adopts the core feature. That fading would give a natural stopping rule, which DiffDiv lacked. Second, we measure artifact reliance directly. We train a linear probe on each head's penultimate features to separate real images from null-edit reconstructions. We also measure disagreement on held-out anchors. With and without the anchor term, this tests the tension that FFR raises against DiffDiv-style mixed training. Third, we study the slow-feature trap from Towards Mitigating Spurious Correlations in the Wild on SpuCoSun-Hard. There, core and spurious features are learned together, and early disagreement may not isolate the shortcut. We expect the method to degrade there and will report that honestly, since a negative result on this benchmark bounds the claim.

Baselines

The naive reference is ERM, meaning plain training that minimizes average loss, on the real data only. On SpuCoSun-Hard it reaches 96% average but 29.3% worst-group accuracy in Towards Mitigating Spurious Correlations in the Wild. On SpuCo Animals standard training reaches 38.5% worst-group accuracy per FFR.

The strongest competitor in the complete-correlation regime is DivDis with its real unlabeled target pool. It reports 82.4% worst-group accuracy on Waterbirds-CC in Diversify and Disambiguate. We run it with the same backbone, head count and disambiguation label budget. Because it uses real target data, it acts as an upper reference, and matching it without real target data counts as success. The strongest generation-based competitor is FFR, which reports 72.6% worst-group accuracy on SpuCo Animals against 54.0% for the prior best synthetic augmentation. We rerun FFR with the same diffusion model at several render budgets, including budgets equal to ours, and give it the subgroup prompts it requires. That knowledge is an advantage our method does not receive. Beating FFR means matching its worst-group accuracy with at least five times fewer rendered images, or exceeding it at equal renders.

We also run DiffDiv. No reported generation-cost figure was found on Lacuna for DiffDiv, so we measure its cost ourselves, including diffusion training, with checkpoint selection done by our own validation rule. On standard Waterbirds, where real counterexamples exist, the reference points are generation-free: data pruning at 90.93% worst-group accuracy from Severing Spurious Correlations with Data Pruning, and DD-DFR at 91.6% with 48 annotations from Dropout Disagreement. We do not expect to beat them there. The claim is that we stay within a few points of them while also working in the regime where they cannot run.

Two ablations isolate the key ingredients. The first is uniform acquisition: identical edits and render budget, with rendered drafts chosen at random. The second removes the anchor term while keeping everything else fixed. Every method gets the same hyperparameter search budget, selected on a held-out validation set with group labels, because the SpuCo benchmark shows worst-group accuracy swinging by over 70% with tuning.

Why not just…?

Why not just generate uniformly at random with the same budget? This is the most serious threat. If caption-swap edits are already mostly shortcut-breaking, acquisition adds little. The uniform-acquisition ablation settles the question directly. If the two compute curves overlap on Waterbirds-CC and SpuCo Animals, the acquisition claim fails, and the contribution shrinks to the anchor term plus the cost accounting. We would report that outcome as such.

Why not just prune hard examples or mine them with dropout disagreement? Data pruning and DD-DFR are cheap and strong whenever the data contain bias-conflicting samples. With complete correlation, however, there is nothing to prune toward and nothing to mine. Our Waterbirds-CC and CelebA-CC experiments are exactly that case. Standard Waterbirds is included so the cost of our method is visible where these simpler methods already suffice.

Why not just use a single model's uncertainty to pick drafts? Entropy from one ERM model shares that model's shortcut. It is confident on counterfactuals that the shortcut labels decisively, which are exactly the images we want. We test this directly by swapping the acquisition score for single-head entropy and single-model Monte Carlo dropout variance, at equal budget.

Why not just use a bigger backbone? The Effect of Model Size on Worst-Group Generalization reports roughly 18.39% lower worst-group error on Waterbirds with larger pretrained models. It also reports plateaus on CelebA and states that the gap is not eliminated. We include ERM with a larger backbone as a baseline so the gains from scaling are separated from the gains from generation.

Why not just interpolate embeddings instead of rendering images? MixPro is far cheaper, but it needs a few real counter-spurious examples to interpolate toward. Under complete correlation those examples do not exist, so embedding mixing cannot create the missing combination. Where a handful do exist, we run MixPro as a low-cost comparison.

Proposed Experiments

The primary benchmarks are Waterbirds-CC and CelebA-CC, the complete-correlation settings used by DivDis, together with SpuCo Animals, CelebA-HQ and UTKFace at 99% to 99.9% bias ratios following FFR. Worst-group accuracy is reported against generator GPU-hours and against rendered-image count, together with average in-distribution accuracy, since DiffDiv reports an in-distribution drop. SpuCoSun-Hard is the stress test.

The decisive checks are three. First, at equal render budget on Waterbirds-CC and SpuCo Animals, disagreement acquisition must beat uniform acquisition by a margin outside seed variance, with at least five seeds. Second, without anchors, the real-versus-synthetic probe accuracy should rise and worst-group accuracy should fall relative to the full method. If neither happens, FFR's artifact concern does not bite in this setting, and the anchor term is unnecessary. Third, our method at its chosen budget should reach FFR's worst-group accuracy on SpuCo Animals with far fewer renders, and FFR is given its subgroup knowledge in that comparison.

A text extension on MultiNLI would replace diffusion with small-model phrase infilling and compare against DISCO's LLM over-generation and teacher filtering pipeline (DISCO). The data-pruning reference there is 75.88% worst-group accuracy. This extension would pursue only the same cost-versus-worst-group-accuracy question.