Do Better Guidance Rules Make Better Training Data? A Pilot-Scale Test

Image generators are increasingly used to produce training sets, not just pictures. A diffusion model is a generator that learns to turn random noise into images by removing noise step by step. Almost all of them are sampled with classifier-free guidance, which pushes each step toward the requested class or caption by a tunable strength. Stronger guidance makes each image cleaner and more on-prompt. It also makes the images more alike. When those images become training data, that sameness hurts. Scaling Laws of Synthetic Images for Model Training ... for Now trained classifiers on 1.3M to 371M synthetic images and measured how fast error falls as the dataset grows. They found that lower guidance gives a steeper curve, yet the best synthetic data still trails real data badly, with a scaling slope of about 0.29 against about 0.5. Over the past two years a family of improved guidance rules has appeared that claims to keep sharpness without losing diversity. None of them has been checked on the quantity that matters for data generation, which is how fast a downstream model improves as more generated images are added.

We propose to measure this directly, and to do so in a way that does not require a 371M-image sweep for every candidate rule. The central bet is that the synthetic-data gap is concentrated in a minority of hard classes, which Fan et al. call the concept bottleneck. If so, a cheap per-class coverage score, computed on a few thousand images per configuration without training anything, should rank guidance rules by their eventual scaling slope. Whole-dataset image-quality scores such as FID should not. If this holds, practitioners get a screening gate: run a pilot of about 10^5 images, discard most guidance configurations, and spend large-scale compute only on the survivors.

Core Claim

We make two claims, and each can be tested separately. The first is empirical. Guidance rules that remove the specific failure that hurts diversity can raise the downstream scaling slope above static guidance at matched prompt adherence. We test two such rules. Adaptive Projected Guidance (Eliminating Oversaturation and Artifacts of High Guidance Scales in Diffusion Models) suppresses the part of the guidance update that acts as a runaway brightness gain. Beta-shaped time schedules (Classifier-free Guidance with Adaptive Scaling) apply guidance mostly in the middle of sampling. Raising the slope is a stronger statement than "lower guidance is better," and it could be false. The projection or the schedule might trade one kind of collapse for a subtler one that shows up only at volume.

The second claim is methodological. Write the downstream error as $L(n) \propto n^{-k}$, where $n$ is the number of synthetic training images and $k$ is the scaling slope. Then a training-free statistic computed on a small pilot predicts the ordering of $k$ across guidance configurations better than FID, Inception Score, or aggregate precision and recall. The statistic is the mean, over classes, of recall of each real class's feature distribution, with the worst classes weighted most. "Better" means higher rank correlation with the confirmed $k$, plus a lower rate of wrongly promoting a configuration through the gate.

Research Program

The work has three parts.

Large-scale reference. We first build a reference set that is large enough to be trusted. We generate class-conditioned ImageNet-1K training sets under roughly a dozen guidance configurations. These are static guidance at several strengths, APG at high strength, beta-schedule guidance, and, on a flow-matching model, the manifold-projected variant from Improving Classifier-Free Guidance of Flow Matching via Manifold Projection. Flow matching is a close relative of diffusion that learns a continuous path from noise to data, and the manifold-projected variant corrects the error introduced when guidance extrapolates linearly between the class-conditioned and unconditioned predictions.

For each configuration we train a fixed downstream classifier from scratch at five dataset sizes, from about 130k up to about 13M images. We fit $k$ with bootstrap confidence intervals over training seeds. This is far below Fan et al.'s 371M images, so we treat our $k$ as a slope estimated within this range, not an asymptote, and say so wherever we report it.

Pilot predictors. Next we compute candidate predictors using only pilot data, about 100 images per class. The proposed predictor embeds pilot images and real ImageNet training images in a frozen feature extractor, computes per-class recall of the real-feature support, and aggregates the scores with emphasis on the lowest classes. The comparison predictors are:

  • FID and Inception Score;
  • aggregate precision and recall;
  • a short-horizon empirical fit of $k$ from downstream training on the two smallest dataset sizes only.

Pilot reliability. Finally, drawing on the preliminary-evaluation literature, we treat the pilot as a noisy decision gate rather than a measurement. We resample pilot images and seeds and report how often each predictor's top-ranked configurations stay in the confirmed top set. The concern is the one raised by Correcting Human Labels for Rater Effects in AI Evaluation: An Item Response Theory Approach, which showed that uncorrected preliminary scores can invert rankings outright. We use no human raters, partly to avoid that failure. The same concern applies to automated proxies, which is why the gate's error rate is reported rather than assumed.

Positioning Against Prior Work

The motivating evidence comes from Fan et al. They established that static guidance strength affects scaling slope and that FID does not track downstream scaling. Their study varied guidance only as one constant per run, and every measurement required a full sweep.

APG, beta schedules, and manifold-projected guidance supply the methods under test. Each paper evaluates single-sample quality, with figures such as FID improving from 10.42 to 6.49 for APG on EDM2-S and Inception Score rising from about 71 to 86 at guidance 2.0 for manifold projection on DiT-XL-2-256. None measures what happens to a model trained on the generated images.

Two papers constrain the design. ManifoldGD projects guidance onto locally estimated data manifolds, but only to shrink datasets to 1–20 images per class. The scalability-gap literature, for example Feng Liu et al., warns that local geometric priors that work in small, clean settings break under heterogeneous data. We therefore include a manifold-projected rule specifically to see whether its benefit survives scaling up, and we do not assume that it will.

Dynamic Classifier-Free Diffusion Guidance via Online Feedback picks guidance strength per step using small reward heads. It optimizes each image's score, and one-image-at-a-time optimization is the mechanism most likely to reduce diversity across a dataset. We include it as a test configuration rather than as a predictor.

Baselines

Real data. The reference ceiling is a classifier trained on real ImageNet images at the same sizes, which gives the real-data slope. Fan et al. report about 0.5 in their supervised setting, against about 0.29 for their best synthetic configuration (Muse with captions). We re-measure both slopes under our own training recipe and do not mix our numbers with theirs.

Naive generation reference. The do-nothing generation reference is static guidance at a moderate default strength, the setting most practitioners would use without thinking.

Strongest existing data-generation method. The strongest existing recipe is Fan et al.'s conclusion: low static guidance, chosen by sweeping. We give it the same tuning budget as the advanced rules, the same number of candidate strengths at pilot scale, with the best one carried to full scale. An advanced rule beats it only if its fitted $k$ exceeds the best low-static $k$ with non-overlapping bootstrap intervals. That improvement must hold at equal or better class-conditional accuracy of a pretrained classifier on the generated images, so a rule cannot win simply by being less on-prompt.

Strongest existing pilot metrics. On the prediction side, the strongest standing metrics are FID and aggregate precision and recall. The proposed statistic beats them if its Spearman correlation with confirmed $k$ across configurations is higher and its bootstrap interval excludes theirs.

Ablation. The key ingredient is per-class worst-case weighting. The ablation computes the same recall statistic averaged over the whole dataset without per-class splitting. If the ablation predicts as well as the full statistic, the concept-bottleneck story is wrong, and we will report that.

Fairness controls. All configurations share the generator checkpoint, the number of sampling steps, the downstream architecture, the training recipe, the evaluation sets, and the number of seeds. We count network evaluations per image, so that methods doubling the cost of sampling are visible.

Why not just…?

Why not just use low static guidance, as Fan et al. recommend? This is a real threat. Fan et al. already showed that lowering static guidance helps scaling, and it may be that no advanced rule beats a well-swept low constant once prompt adherence is matched. The evidence against it as a final answer is limited. It still leaves the 0.29 versus 0.5 gap and the concept bottleneck, and the manifold-projection paper shows that low static guidance gives weak class separation (Inception Score about 71 at scale 2.0). Weak class separation may matter for class-conditional training data. Our tuned low-static baseline settles this directly. A null result here is a publishable answer to Gap 1, not a failure.

Why not just fit the scaling slope from two small training runs? This is the most dangerous alternative to the predictor, because it measures the target quantity directly, only at small scale. Its weaknesses are cost and noise. It still requires downstream training per configuration, and a slope fitted from two points has high variance. Its strength is that it cannot be fooled by a feature extractor's blind spots. We include it as a full competitor. If it matches the training-free statistic's rank correlation at similar cost, we will recommend it and recast the contribution as the scaling study plus a validated cheap protocol.

Why not just screen with FID or Inception Score? Fan et al. report that these single-image quality scores do not track downstream scaling efficiency. APG's and manifold projection's headline gains are exactly in FID and Inception Score, which is why it is unknown whether those gains transfer to training data. We measure FID's rank correlation with confirmed $k$ explicitly, so this claim is tested rather than inherited.

Why not just use per-step online feedback to pick guidance? Per-image reward maximization improves each image's score. Under the overlap analysis in Calibrated Test-Time Guidance for Bayesian Inference, reward-maximizing guidance collapses variance, which is the opposite of what a training set needs. Including Dynamic CFG as one generation configuration tests this. We predict a lower $k$ than its per-image win rates (53.8% on GenAI-Bench with Imagen 3) would suggest.

Proposed Experiments

Generators and data. We use class-conditional DiT-XL/2 and EDM2 for ImageNet 256×256, and a flow-matching model for the manifold-projected variant. Each configuration produces up to about 13M images.

Downstream training and evaluation. The downstream model is a ResNet-50 trained from scratch on synthetic images only. We evaluate it on the real ImageNet validation set and on ImageNet-A, -R, -Sketch, and -v2, because Fan et al. found that synthetic data's robustness advantage grows with scale. We also check a CLIP-style variant at small scale against the 15-dataset zero-shot protocol, where Fan et al. report average accuracy rising from 9.9% to 16.5% when synthetic data is mixed with real.

Metrics. The metrics are the fitted $k$ with bootstrap intervals, the per-class error curves, Spearman correlation between each pilot predictor and confirmed $k$, and the gate's false-promotion rate under pilot resampling.

Decisive checks. Three outcomes decide the result.

  1. Does any advanced rule's $k$ exceed the tuned low-static baseline's at matched adherence?
  2. Does the per-class pilot statistic rank configurations better than both FID and the short-horizon fit?
  3. Does the ablation without per-class weighting lose that advantage?

Further, if the per-class statistic is right, the classes it flags as worst covered should be the classes whose error curves are flattest at full scale. This prediction can be checked class by class and could fail cleanly.