Spending the Diffusion Budget on Sampled Regions, Not Whole Images

Training image classifiers on extra pictures made by a diffusion model (a generator that turns random noise into an image by denoising it step by step) helps. It is also expensive. Most of that cost goes into regenerating pixels the classifier already handles well. Recent work cuts the bill by choosing which images to augment. This proposal argues that the useful unit of choice is a region inside an image. We use the classifier's own heat map of where it looks, raise it to a temperature, and sample from it to decide which regions to repaint. A diffusion inpainting model then repaints only a cropped window around each sampled region. Per-sample cost then shrinks roughly with the window's area. The sampling keeps the generation budget from piling onto the one patch the model already fixates on, such as the beak of a bird.

The idea carries a known failure of greedy spatial selection into data generation. In discriminative vision, always picking the top-scoring region makes a network shrink its attention onto one narrow cue, and sampling regions in proportion to their scores fixes this. We expect the same failure when a generation budget is spent greedily. The temperature knob makes that claim testable, because it moves the method continuously from uniform-random regions to argmax regions.

Core Claim

We make three claims for a fixed budget of diffusion GPU-hours.

The first is that allocating generation over pairs of (image, region), rather than over whole images, gives higher downstream accuracy. Synthesizing a crop costs less than synthesizing a full canvas, so the same budget buys many more distinct variations.

The second is that the right allocation samples regions with probability rising with the classifier's attribution, at an intermediate temperature. Concretely, a candidate region $r$ of image $x$ is drawn with probability

$$p_\tau(r \mid x) \propto a(r)^{1/\tau},$$

where $a(r)$ is the classifier's class activation map (a per-location score of how much each place drives the predicted class) summed over $r$, and $\tau > 0$ is a temperature. As $\tau \to 0$ this becomes argmax selection; as $\tau \to \infty$ it becomes uniform random regions.

The third is that both endpoints lose to an intermediate $\tau$. Argmax loses because it regenerates the same shortcut region again and again. Uniform loses because it wastes budget repainting background the model ignores.

If the experiments show that uniform regions ($\tau \to \infty$) do as well, the second claim fails. What would remain is a narrower compute claim about crop-level inpainting, and we would report it that way.

Prior Work and What It Leaves Open

Region sampling. The regional mechanism comes from discriminative work showing that sampling beats greedy selection. Stochastic Pooling replaces max-pooling with sampling in proportion to activation. Importance Sampling CAMs samples training pixels from the activation map so that the whole object lights up rather than its most discriminative core. Stochastic-based Patch Filtering samples ViT patches from a softmax over class similarity instead of keeping the top-$K$. It reports that the retention ratio needs careful tuning, with 50% best on food benchmarks. Saliency Grafting shows that greedy maximum-saliency mixing (PuzzleMix) lacks diversity and that labels must be calibrated to what was actually sampled. All four operate on existing pixels or features. None decides where to spend generation compute, which is the step we add.

Budget targeting. Three papers motivate and constrain the work on the generation side. TADA finds hard "slow-learnable" images by clustering early-training losses and re-diffuses only those. That cuts generation to about 0.3–0.4× of full augmentation, but noise is still added to the full canvas. Difficulty Controlled Diffusion conditions on a single image-level difficulty score and finds a middle "Goldilocks" band works best. That finding is a global analogue of our intermediate-temperature hypothesis. Background augmentation inpaints only the background of detection images and scales denoising steps with the masked area. It is the closest spatial precedent, but its mask is a fixed foreground/background split taken from ground-truth boxes, and it regenerates all background pixels. Our method differs from all three in one respect: the spatial mask is sampled from the learner's own attention, and compute is spent only inside the sampled window.

Constraints. Two results limit the design. Growing Images proves that progressive, region-by-region denoising schedules fall into locally consistent but globally wrong states, raising FID 2–5× on CelebA. We therefore never chain regions sequentially. Each sampled window is denoised in parallel in one pass, with its surrounding context visible, and each window yields its own independent sample. AugGen reports that FID correlates poorly with downstream recognition. For that reason we judge success by downstream accuracy per GPU-hour, never by image-quality scores.

Post-hoc curation. SDS filters full generated images after the fact, discarding about 35% of a 40k pool. Its wasted generation compute is the cost we try to avoid by deciding where to generate before generating.

Research Program

Warm-up and attention maps. The pipeline begins with a short warm-up training run on real data, as in TADA. We then compute activation maps for every training image; this is a forward pass and costs little next to diffusion. The maps are refreshed once mid-training so the sampler follows how the model's attention moves.

Region candidates. Candidates are square windows at two or three scales on a grid. For each draw we sample a window from $p_\tau$ and crop a context margin around it. We then run a latent-diffusion inpainting model on the crop with a class-conditioned prompt and paste the result back.

Choosing which images to sample. Images are drawn either uniformly or from TADA's slow-learnable cluster. This lets us ask whether region-level and image-level targeting add up.

Label safety. Repainting the discriminative region could make an image look like another class, a real risk in fine-grained categories. Every synthesized image is therefore checked by the warm-up classifier together with a CLIP prompt check, and rejected if both disagree with the label. The cost of this check counts toward the budget.

Compute accounting. We count generation cost as

$$C = \sum_i \frac{|w_i|}{|x_i|}\, S_i\, c_{\text{full}},$$

where $|w_i|/|x_i|$ is the crop's share of the image area, $S_i$ is the number of denoising steps, and $c_{\text{full}}$ is the per-step cost on a full canvas. The linear dependence on area holds because convolutional and latent-attention costs scale roughly with the number of latent tokens processed. We also measure wall-clock time directly, because fixed overheads and minimum resolutions make small crops cost more than linear scaling predicts.

Analysis. Beyond accuracy, we track how spread out the classifier's attention is: the entropy of its normalized activation map, and the share of the object box the map covers. This directly tests whether sampled-region generation prevents the winner-take-all contraction seen in the discriminative literature.

Baselines

Naive reference. Standard augmentation (random crop, flip, color jitter) with zero generation compute. A plain generation reference sits next to it: full-image re-diffusion of uniformly chosen images at the same budget. Every curve is plotted as accuracy against generation GPU-hours. A method "beats" another only if its curve is higher at matched budget.

Strongest whole-image competitor: TADA. TADA is the strongest method on exactly this question, compute-efficient targeted diffusion augmentation for classification. It reports up to +2.8% accuracy across CIFAR-10/100, TinyImageNet and others, and on ImageNet-1K it beats Boomerang with 65% of the augmentation budget. We reimplement it with the same inpainting backbone used in full-canvas mode, the same warm-up classifier, the same student architectures, and the same budget grid. Its noise level gets the same tuning sweep as our temperature. Beating it means higher accuracy at equal GPU-hours on at least three of the four fine-grained datasets, or equal accuracy at clearly lower cost.

Image-level difficulty conditioning. Difficulty Controlled Diffusion is the second strongest competitor. It reports beating Real-Fake by over 1.2% on Stanford Cars and saving 63.4 GPU hours on ImageNet with 10% extra synthetic data. Its LoRA fine-tuning cost is counted in its budget. If reimplementation proves infeasible, we compare against its reported Stanford Cars setting with matched data volume and say so.

Detection. The baseline is background augmentation, which reports +2.7 mAP in the 10% COCO regime and +5.3 at 10× augmentation. We run it in the same 10% COCO regime with the same detector, at matched GPU-hours.

Zero-generation regional mixing. Saliency Grafting reports 13.05% CIFAR-100 Top-1 error against 14.47% for CutMix. It costs no generation compute, so it marks the point any generative method must clear to justify its cost. We also run it together with our method.

Temperature ablation. The ablation that isolates our key ingredient sweeps $\tau$ from near-zero (argmax region) through intermediate values to infinity (uniform region). Everything else stays fixed: the same crop sizes, the same verifier, and the same budget.

Why not just…?

Why not just use random regions? This is the most serious threat. Random-window inpainting already gets the crop-level compute savings, and diversity alone might be what matters. The $\tau \to \infty$ arm of the sweep tests exactly this. If it matches the intermediate temperature within seed noise on the fine-grained sets, we drop the attribution claim. Two pieces of evidence favor attribution weighting. CutMix loses to attribution-guided sampling in Saliency Grafting. And the "context over content" gains in background augmentation suggest that where you repaint matters. Neither result is about generated pixels, though, so the question stays open until we run it.

Why not just take the top-scoring region deterministically? Greedy regional selection loses diversity: PuzzleMix scores 14.78% error against 13.05% for Saliency Grafting, and patch filtering reports top-$K$ locking onto fixed configurations. In generation the problem may be worse, because the same beak would be repainted repeatedly. The $\tau \to 0$ arm measures this, and the attention-entropy analysis shows whether the model's attention shrinks under it.

Why not just use TADA's whole-image targeting, or simply generate everything? Brute-force 10–30× expansion causes noise magnification and wastes compute on easy data (TADA; Difficulty Controlled Diffusion). TADA itself is the real competitor, and the matched-budget curves decide it. We also run TADA-selected images with region-level sampling, which tests whether the two kinds of targeting add up. If they do, the contribution is a complement to TADA rather than a replacement.

Why not just use free augmentation and skip generation? The comparison points the other way here: DreamDA lifts ResNet-50 on Pets from 43.8% to 86.5% over standard training, which shows generation can add variation that cropping and mixing cannot. Saliency Grafting stays in every table as the zero-cost bar.

Proposed Experiments

Main experiments. The main testbed is fine-grained classification on CUB, Stanford Cars, Flowers102 and Oxford-IIIT Pets, with ResNet-50 and ViT-S students. In this setting a single-cue shortcut is most likely and the cost of repainting the wrong region is highest. No reported result was found on Lacuna for TADA or Difficulty Controlled Diffusion on CUB or Flowers102 specifically, so those numbers come from our reimplementations. Budgets run from 0.1× to 0.5× the GPU-hours of TADA's full augmentation, with three seeds per point.

Secondary experiments. We test transfer to CIFAR-100 and to the 10% MS COCO detection regime. For detection, sampled windows may overlap objects only partially, and boxes are kept only where the verifier confirms the object survives.

Decisive checks. The first is the $\tau$ sweep: the hypothesis requires an interior optimum. The second is matched-GPU-hour comparison against TADA. The third is attention entropy and box coverage, which should be higher for the intermediate temperature than for argmax. The fourth is a seam-artifact control: train on crops pasted back without any repainting, to make sure gains are not coming from edge artifacts. Following Growing Images, we also run a sequential multi-region variant to confirm that parallel single-window repainting avoids the drift it predicts. If the attribution-weighted sampler wins on both the budget curve and the attention analysis, generation compute should be allocated by sampling where the learner looks, not by choosing images.