Deciding Early Which Synthetic Images to Finish

Training image classifiers on pictures made by diffusion models (generators that turn random noise into an image by removing noise over many small steps) helps most when real data is scarce. It is also expensive. One handwriting study needed 62 hours on 8 high-end GPUs just to produce its synthetic training set. The standard way to make synthetic data useful makes the cost worse: generate a large surplus, then throw most of it away or down-weight it. Every discarded image still paid for the full denoising chain.

This proposal asks whether that decision can be made partway through sampling instead of at the end. Diffusion models commit to an image's semantic content surprisingly early. So we can stop a trajectory after a small fraction of its denoising steps, look at the model's current guess of the final image, and ask a cheap scorer whether finishing it would help the downstream classifier. Only the promising trajectories are completed. The claim to test is concrete: at a matched total compute budget, counting generation, selection and classifier training together, pruning trajectories mid-flight beats both fast samplers with post-hoc selection and difficulty-conditioned generation.

Core Claim

Existing synthetic-data selectors act on fully generated pools. Covariance Matching greedily picks finished samples whose feature covariance matches the real data (High-dimensional Analysis of Synthetic Data Selection). Informative Data Selection re-weights finished chest X-rays with a learned network (Informative Synthetic Data Generation for Thorax Disease Classification). Diffusion analysis, meanwhile, shows that the clean-image prediction $\hat{x}_{0,t}$ at an intermediate step reveals semantic flips far earlier than pixel differences do (Early Semantic Commitment in Diffusion Sampling). That work is purely diagnostic and never terminates anything.

Our claim is that a selection score computed on $\hat{x}_{0,t}$ at a checkpoint early in sampling ranks trajectories by downstream usefulness well enough that pruning there dominates post-hoc selection on the cost–accuracy curve.

The savings are easy to state. Suppose we launch $N$ trajectories, check them after a fraction $f$ of the sampling steps, and continue a fraction $q$ of them. The generation cost per retained sample, relative to fully generating all $N$ and keeping the same $qN$ afterward, is

$$ \frac{C_{\text{in-flight}}}{C_{\text{post-hoc}}} = \frac{f + q(1-f)}{1}. $$

This follows because every trajectory pays for $f$ of the chain and only survivors pay for the rest. For example, with $f=0.2$ and $q=0.25$ the ratio is $0.4$, roughly 2.5 times cheaper generation.

The catch is that the ranking gets noisier as $f$ shrinks. There is therefore an optimal checkpoint, and it likely differs across criteria. Class identity settles early, but the fine-grained cues that make a sample "medium difficulty" may settle late. Mapping this tradeoff is the scientific core of the proposal.

How It Relates to Prior Work

Three lines of work motivate the idea.

  • Informative samples matter more than many samples. Difficulty Controlled Diffusion (Difficulty Controlled Diffusion Model for Synthesizing Effective Training Data) shows that unguided generation floods the pool with easy samples. Its approach is to condition generation on a difficulty score from an external classifier, via a LoRA-fine-tuned difficulty encoder. It still runs every trajectory to completion. We instead select among cheap unconditioned trajectories, with no generator fine-tuning.
  • Post-hoc selection helps. Covariance Matching supplies our diversity criterion, which we reuse. We move it onto previews and make it incremental.
  • Budgets must count everything. Compute-Constrained Data Selection showed for language-model training that expensive selectors often lose to near-free ones once selection compute is counted. This constrains our design: the scorer must be tiny relative to the generator. It also shapes our evaluation, which extends their budget to include generation.

Two sampling papers are method neighbors, and they differ in objective.

One finding cuts against naive pruning. Gui et al. found that messier, low-guidance handwriting samples trained better recognizers than clean ones (Zero-shot Generation of Training Data with Denoising Diffusion Probabilistic Model for Handwritten Chinese Character Recognition). A scorer that prunes "ugly" previews could therefore remove exactly the useful samples. For this reason, our scorer never rewards visual quality. It rewards only two things: label consistency, and marginal contribution to difficulty and diversity.

Research Program

The pruned sampler. We use a pretrained text-to-image generator (Stable Diffusion 1.4 or PixArt-α, the generators used in the Covariance Matching study) with a fast deterministic solver of about 20–50 steps. Trajectories run in large batches. At a checkpoint fraction $f$, we compute $\hat{x}_{0,t}$ for each trajectory and embed it with the current student classifier's encoder. This forward pass is negligible next to the generator.

The scorer combines three signals:

  • Label consistency. Does the student's prediction on the preview agree with the prompted class? This rejects semantic drift.
  • Difficulty. One minus the student's confidence in the true class, with an upper cap to avoid atypical outliers. This follows the medium-difficulty band found by Difficulty Controlled Diffusion.
  • Diversity. The greedy reduction in the Frobenius distance between the covariance of the retained set and the covariance of the real data, measured in a 32-dimensional PCA space as in Covariance Matching.

Using the student as the scorer removes the external teacher. It also makes selection adaptive: what counts as hard changes as the student learns. Generation and training therefore alternate in rounds.

A reliability study before the full system. For each benchmark, we fully generate a pool and record previews at many checkpoints. We then measure how well each preview-based score (label, difficulty, covariance gain) at checkpoint $f$ predicts the same score on the finished image, using rank correlation and top-$q$ overlap. This identifies where each criterion becomes reliable, which is the empirical content of the $f$ tradeoff. It also connects to the early-commitment findings, which were measured only on binary face attributes and unconditional generation.

Budget accounting. All methods are charged in measured GPU-hours and FLOPs for:

  • generator fine-tuning, if any;
  • sampling evaluations, including those spent on pruned trajectories;
  • scoring;
  • student training.

Results are reported as accuracy-versus-total-budget curves, not at a single point.

Baselines

Real data only, with standard augmentation. This is the do-nothing reference. It sets the floor that any synthetic pipeline must beat to justify its cost, and it gets the same student architecture and tuning budget.

Fast solver, full generation, random subset. This is the naive synthetic reference and the budget-matched version of "just generate more." Its full-generation analogue is informative: Informative Data Selection reports that unfiltered synthetic data dropped CheXpert mAUC from 89.3% to 89.0%.

Fast solver plus post-hoc Covariance Matching. This is the strongest competitor on the selection axis, because it uses the same diversity criterion on finished images. The paper reports 54.00% CIFAR-10 accuracy from scratch, versus 49.38% for random selection and 50.04% for center matching (High-dimensional Analysis of Synthetic Data Selection). We rerun it with the same generator, solver, student and training schedule, and we give it whatever pool size the budget allows. Beating it means reaching higher accuracy at equal total GPU-hours, or equal accuracy at clearly lower cost, across at least three budget levels.

Difficulty Controlled Diffusion. This is the strongest competitor on the targeted-generation axis. It reports peak ImageNet accuracy with only 10% extra synthetic data, saving 63.4 GPU hours relative to Real-Fake, and more than 1.2% higher Stanford Cars accuracy than Real-Fake. The file gives no absolute accuracy values for these results, so we will reproduce it on Stanford Cars, CUB and EuroSAT. Its LoRA fine-tuning, and its classifier scoring of the real set, are charged to its budget.

Post-hoc pruning with our scorer. This is the key ablation. It uses the identical scorer applied only to finished images, so it isolates the in-flight ingredient. We also run random pruning at the checkpoint, which isolates the scorer.

Our method combined with layer skipping. This tests whether the gains compound with the early-exiting method.

Why not just…?

Why not just use a faster solver and select afterward? This is the most serious threat. If sampling takes 10 steps, the absolute savings from pruning shrink, and post-hoc Covariance Matching on a cheaply generated pool may match us. The savings ratio above still applies at any step count. However, fewer steps may also make previews at a fixed $f$ less reliable. The comparison against the post-hoc Covariance Matching baseline, run at every solver step count we test, settles this directly. If we lose at 10–20 steps, the honest conclusion is that in-flight pruning pays off only for expensive generators, and we will report it that way.

Why not just condition the generator on difficulty? Conditioning avoids waste without discarding anything. It does, however, require fine-tuning the generator and an external scorer. It also fixes difficulty relative to a frozen classifier rather than the evolving student. The two approaches are complementary, so we also run pruning on top of the difficulty-conditioned generator. If conditioning alone wins at equal budget, pruning adds nothing.

Why not just use traditional augmentation or more real data? In the regimes targeted here, extra real data is not available. Examples are fine-grained classes, medical imaging, and rare characters, where diffusion reached 93.0% zero-shot accuracy on unseen handwritten characters that standard augmentation cannot produce. The real-data-only baseline keeps this claim honest on each benchmark.

Why not just make each step cheaper with early-exit layers? Layer skipping cuts the cost per step by 25–30%, but it still finishes every discarded image. It has also never been evaluated as a source of training data. We test it alone and combined with our method; the pruning savings should multiply its gains rather than replace them.

Proposed Experiments

Benchmarks. We use CIFAR-10 (training from scratch, matching the Covariance Matching protocol), Stanford Cars, CUB, EuroSAT, and Imagenette or an ImageNet-100 subset. As a transfer test in a domain where real data is scarce, we use CheXpert with a latent diffusion generator, compared against Informative Data Selection's reported 90.1% mAUC.

Metrics. We report top-1 accuracy (mAUC for CheXpert) as a function of total GPU-hours, and the area under that curve. For the preview study, we report rank correlation between preview scores and finished-image scores at each checkpoint.

Decisive checks. The proposal fails if any of the following holds:

  • Preview scores at $f \le 0.3$ show weak agreement with finished-image scores, measured as top-quartile overlap barely above chance.
  • In-flight pruning does not beat identical-scorer post-hoc pruning at matched budget.
  • Fast-solver-plus-Covariance-Matching dominates our curve at every step count.

We also directly examine the handwriting-style concern: whether pruned-away trajectories are disproportionately the messy, low-guidance ones, and whether reinstating them helps.