Refresh or Realize? Compute Allocation in Drifting Models

Sipeng Chen, Xu Zheng, Shibo LiarXiv 2026 · arXiv

Generative modeling has seen a surge of "one-step" generators, with Drifting Models being a prominent recent entry. These models work by calculating a "drift field" that tells generated samples how to move toward the real data distribution. But there's a catch: because the model's weights are shared across all samples, a single optimization step can't perfectly follow the prescribed motion for every sample. This creates a fundamental training question: should you spend your limited compute budget on fitting the current target more accurately (realize), or on recomputing a fresh target field (refresh)?

A new paper, "Refresh or Realize? Compute Allocation in Drifting Models," dives deep into this trade-off, revealing that when it comes to wall-clock efficiency, "freshness" beats "depth."

TL;DR

Drifting Models face a choice between fitting a stale target field better or recomputing a fresh one. This paper demonstrates that while deeper fitting improves realization, refreshing the field with new data is a far more efficient use of compute, leading to better image quality (lower FID) for the same wall-clock time.

What Problem Does This Paper Address?

Drifting Models train a generator by recomputing a finite-sample drift field at every iteration. Ideally, the generator would move samples exactly along this field. However, in practice, the move happens in parameter space. Since one set of weights is shared by the whole batch, the actual motion is a "tangent-filtered" version of the prescribed one.

The realization gap: one optimizer step (red) doesn't perfectly follow the prescribed motion (blue). Taking more steps (orange) gets closer but never perfect.

This leaves researchers with a compute allocation problem. Is it better to take multiple optimizer steps ($k > 1$) to "realize" the current target better, or to take just one step and then "refresh" the target with new data?

Methods

The authors study this on ImageNet 256x256 and toy datasets like the Swiss-roll. Their methodology includes:

  • Frozen-Target Probes: Freezing a single target field and measuring how $k$ optimizer steps move the network closer to it.
  • Realization Diagnostic: A new metric that decomposes network progress into aligned gain (useful motion toward the target) and orthogonal penalty (unintended motion).
  • Resource-Matched Comparisons: Comparing training runs with $k=1$ vs. $k=2$ steps per refresh under two conditions: equal number of refreshes (where $k=2$ uses more total compute) and matched wall-clock time.
  • Local Allocation Probes: Controlled experiments at specific checkpoints to compare taking a second step toward a stale target vs. a first step toward a fresh one, using exactly matched compute.

Datasets

  • ImageNet 256x256: The primary benchmark for high-quality image generation.
  • Swiss-roll & Checkerboard: 2D toy datasets used for visualizing the realization gap and training dynamics.

Key Insights and Ideas

  • The Realization Gap is Measurable: Shared parameters act as a filter. Early in training, it can take up to 16 steps to make any net progress toward a target field because the "orthogonal penalty" outweighs the "aligned gain."
  • Support Resampling Dominates Target Change: The target field is built from a finite sample of data. The authors found that drawing a new sample (refreshing) rotates the target direction significantly (cosine similarity $\approx 0.3-0.6$). In contrast, a parameter update barely changes the target's direction (cosine $\approx 0.95-0.98$).
  • Staleness is Noise-Fitting: Because a stale target barely moves in direction, taking extra steps toward it mostly fits the specific noise of that finite data sample rather than the underlying distribution.
  • Field-Space Optimality $\neq$ FID Gains: Interestingly, making the target field more "optimal" in a mathematical sense (e.g., through better estimators) doesn't reliably translate to better final image quality (FID).

Key Results

  • The Refresh-Realize Reversal: If extra compute is "free" (equal number of refreshes), taking two steps ($k=2$) lowers FID. However, once you pay for the compute (matched wall-clock time), the result flips: spending the budget on more refreshes ($k=1$) reaches lower FID on both training seeds.
  • Universal Local Win: In all 40 local probes across different training stages and support sizes, refreshing the target outperformed taking a second step toward a stale one when compute was matched.
  • Realization Improves with Training: As training proceeds, the network becomes "easier" to move. The depth needed to make net progress falls from $\approx 16$ steps at the start to just 1 step by checkpoint 20k.

Limitations and Future Work

The authors acknowledge that their findings are specific to the ImageNet regime and the particular affinity constructions used in current Drifting Models. They don't claim an ordering for arbitrary realization depths (e.g., $k=100$).

Future work could investigate whether different affinity designs or optimization strategies (like second-order methods) might shift the balance back toward deeper fitting. For now, the practical advice for Drifting Model practitioners is clear: if you have extra time, spend it on fresh data, not deeper fitting.

Summary updated

Abstract

Drifting Models train a one-step generator by recomputing a finite-sample drift field at every iteration and taking an optimizer step toward the drifted target. The field says how generated samples should move, but the step is taken in parameters shared by all samples, so the motion the network actually makes need not match the motion it was given. This leaves a basic training question open: should extra compute go into fitting the current target more closely, or into recomputing the field? We study it on ImageNet 256x256. Holding the target fixed for k optimizer steps and measuring the realized displacement, we find that deeper fitting does bring the network closer to the frozen target, and that the number of steps needed before it makes any net progress drops from about sixteen early in training to one later on. When the extra steps come for free, k=2 also lowers FID. Once they are paid for, the result flips: at approximately matched measured wall-clock, spending the budget on fresh fields gives lower FID than deeper fitting, on both training seeds. The target itself shows why a fresh field is worth so much. Redrawing the finite support rotates its direction far more than a parameter update does (cosine ~0.3-0.6 against ~0.95), and a correction that is optimal in field space is not reliably better in FID than a parameter-free one. For Drifting, fitting each target well and spending compute well are different goals.
BibTeX
@misc{chen2026refresh,
  author = {Sipeng Chen and Xu Zheng and Shibo Li},
  title = {{Refresh or Realize? Compute Allocation in Drifting Models}},
  year = {2026},
  eprint = {2609.32298},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.32298},
}
Figure