Iterative refinement repairs a generated output by repeatedly editing it until it looks right. Its cost grows with every pass. Iterative Partial Refinement, a diffusion method that repeatedly picks image regions, adds noise to them, and regenerates them, raises the hard-Sudoku solution rate from 55.8% to 75.0%. It needs 50 passes to get there, roughly 13 times the compute of one generation, and it chooses which regions to redo at random (Inference-Time Scaling in Diffusion Models through Iterative Partial Refinement). Most of that compute is spent regenerating parts that were already fine, and on samples that were already finished. The authors themselves call for informed region selection and adaptive stopping.
This note proposes a cheap source for both decisions. A batch normalization (BN) layer rescales intermediate activations using a per-channel mean and variance that it recorded on the training data and then froze. Those stored numbers form a small Gaussian summary of what normal features look like at each depth. Prior work shows that a test input whose activations depart from them is likely out of distribution. We propose to run each intermediate draft through a frozen BN network trained only on clean in-distribution samples, and to read off a per-location departure map. Peaks in the map say where to re-noise. When the whole map falls below a threshold calibrated on clean data, refinement stops. The monitor needs one forward pass, generates no text, and needs no anomaly examples.
Core Claim
The claim has two parts, and both can be checked directly. First, the spatial departure of a draft's activations from stored BN statistics is concentrated on the locations that actually violate the task's constraints. Second, using this map to choose which regions to re-noise, and its global maximum to decide when to stop, lets partial refinement reach its 50-iteration quality in a fraction of the iterations. It should also cause fewer cases where an already-correct output is edited into an incorrect one.
To make this concrete, let $a_{\ell,c}(u)$ be the activation of channel $c$ at spatial location $u$ in layer $\ell$ of the monitor network. Let $\mu_{\ell,c}$ and $\sigma^2_{\ell,c}$ be that layer's frozen BN running mean and variance, and let $C_\ell$ be its number of channels. The per-location departure score is
$$d_\ell(u) = \frac{1}{C_\ell}\sum_{c=1}^{C_\ell} \frac{\big(a_{\ell,c}(u)-\mu_{\ell,c}\big)^2}{\sigma^2_{\ell,c}},$$
and the map $D(u)$ is the sum over a chosen set of layers of $d_\ell(u)$, upsampled to the input resolution. Regions are sampled for re-noising with probability proportional to $D$. Refinement halts when $\max_u D(u)$ drops below the 95th percentile of the same statistic computed on clean training samples.
That calibration uses only clean data. It sidesteps the synthetic-outlier problem shared by earlier BN detectors: pixel permutation in NMD, uniform noise in 1C-Sum, and adversarial perturbations in SAFE.
Prior Work and Positioning
Two groups of papers motivate the idea by showing that BN departures carry anomaly signal.
Neural Mean Discrepancy for Efficient Out-of-Distribution Detection classifies inputs from the gap between live activation means and stored BN means. It reports AUROC often above 99% at 7.54 ms per example. Its future-work section suggests using layer-wise discrepancies for early exits, but the paper never builds any stopping mechanism. Sample-Free White-Box Out-of-Distribution Detection for Deep Learning formalizes departure-from-mean and departure-from-standard-deviation scores that need only forward passes. Our per-location score is a spatial version of those scores.
Both papers score whole images. Two others show that BN-anchored signals can also localize anomalies. SAFE pools BN-equipped residual-block features over candidate object boxes and cuts the false-positive rate at 95% recall (FPR95) on OpenImages from 48.3% to 17.7%. BatchNorm-Based Weakly Supervised Video Anomaly Detection scores video snippets by their Mahalanobis distance from the BN mean. Neither paper feeds its scores back into a generator.
Online use of BN statistics as a trigger has precedent. Effective Restoration of Source Knowledge in Continual Test Time Adaptation watches the divergence between test-batch and source BN statistics and resets the model when it spikes. We borrow that "monitor and act" pattern, but act on a generator's edit loop rather than on model weights.
Two papers constrain the design.
GraFIQs shows that backpropagating the BN mismatch yields a sharper quality signal. That signal costs a backward pass on every iteration, which defeats the purpose here, so we test it only as a costlier comparison. Instance-dependent Early Stopping stops training on each sample separately once its loss curve flattens, and shows that uniform budgets waste compute because samples differ in difficulty. It operates at training time; we apply the same per-sample intuition at inference.
One scope decision deserves stating plainly. The obvious target, the critique-and-revise loop in Volcano, roughly doubles latency, from 2.7 s to 5.8 s per response. But its backbone is a vision transformer and a language model, which use layer normalization rather than batch-level running statistics. That leaves no stored training-set footprint to compare against. We therefore choose the image-space diffusion refinement setting, where a convolutional BN monitor applies naturally. We treat language-model loops as outside the claim.
Research Program
We first reproduce Iterative Partial Refinement on its three constraint benchmarks: MNIST Sudoku (hard split), Counting Polygons, and Even Pixels.
For each benchmark we train a monitor, a BN ResNet, on the same clean training images the diffusion model saw. On MNIST Sudoku the monitor is a cell-level digit classifier applied over the board. On the other two benchmarks the monitor learns a pretext task, rotation prediction, because BN running statistics reflect the training data regardless of which objective produced them.
We then analyze the maps before using them for anything. Sudoku has an exact rule checker, which tells us which cells participate in row, column, or box conflicts. Polygons and Even Pixels have programmatic checks for vertex counts and pixel balance. These checks give ground-truth "bad regions," and we measure how well $D$ localizes them, layer by layer.
There is a real risk here. Sudoku violations are relational: a repeated digit is a perfectly normal-looking digit. A local convolutional monitor may therefore miss them entirely. Under this risk, the map would mostly flag poorly rendered digits, not logical conflicts. The layer analysis exists to find out which of these the map flags.
Only after this check do we insert the map into the refinement loop. We compare three versions: guided selection alone, the halting rule alone, and both together. For each we trace the full curve of quality against compute, measured in denoiser passes plus monitor passes.
Baselines
The naive reference point is Iterative Partial Refinement exactly as published: random region selection and a fixed 50 iterations. Its reported results, from Inference-Time Scaling in Diffusion Models through Iterative Partial Refinement, are as follows:
- Hard Sudoku valid-solution rate rises from 55.8% to 75.0%.
- Polygon number-match accuracy rises from 15.4% to 27.4%, and vertex uniformity reaches 100%.
- Even Pixels balance accuracy improves by about 5%.
- All of this costs about 13 times the compute of a single generation pass.
We rerun this baseline ourselves under identical seeds so that every comparison is matched. Beating it means reaching 75.0% on Sudoku with substantially fewer total passes, with a target of half or less, without a lower final rate.
The second baseline is the same method with a fixed, smaller iteration budget, swept from 5 to 50. This traces the random-selection quality–compute curve. Our method must lie above that curve at equal compute, not merely beat the single 50-iteration point.
The strongest competitor is a self-consistency map computed from the diffusion model itself. Each region is re-noised lightly and denoised once, and the reconstruction error is recorded; high error marks a region the model itself considers implausible. It needs no extra network, which makes it the most natural alternative. It does cost extra denoiser passes per iteration, and we charge those passes to its compute budget. We tune its noise level on the same validation split and with the same number of trials as our layer and threshold choices. To our knowledge Lacuna reports no result for any informed region selection in this setting, so this baseline is built here.
The ablation that isolates the key ingredient keeps the same monitor network but replaces the BN-statistic score. The replacement is a per-location Mahalanobis distance to feature means that we re-estimate ourselves from held-out clean data with full covariance. If this ablation matches our method, the value lies in any feature-density monitor, and the stored BN statistics merely supply a free copy of it. That is a weaker but still useful claim.
For halting, we compare against the maximum softmax probability of the same monitor. We also compare against a GraFIQs-style gradient-magnitude score, which costs one backward pass per iteration. Finally, the exact rule checker serves as an oracle upper bound for both where to refine and when to stop.
Why not just…?
Why not just use the diffusion model's own reconstruction error to pick regions? This is the most serious threat. It may well localize low-level artifacts as well as the BN map does, and it needs no second network. What counts in its favor or against it is cost and coverage. Each self-consistency probe adds denoiser passes, while the monitor adds one small CNN forward pass. If the self-consistency map matches ours in localization and in quality per unit of compute, we would report the BN monitor as unnecessary for this setting. The localization analysis and the compute-matched curves settle this directly.
Why not just stop when the monitor or the generator is confident? BOWLL documents an "inverted entropy" failure in which predictive entropy does not separate corrupted inputs from clean ones, while internal BN statistics do. SAFE similarly finds that output heads fail silently on out-of-distribution objects. These results concern classification inputs, not generated drafts, so we do not assume they transfer. The softmax-halting baseline tests this on our own benchmarks.
Why not just run fewer iterations? The published scaling curves show Sudoku and polygon accuracy still climbing out to 50 iterations, so a uniform cut sacrifices hard instances. Instance-dependent Early Stopping gives the general argument that per-sample budgets dominate uniform ones. The budget sweep makes this concrete: the halting rule must beat every point on the fixed-budget curve, not just the 50-step endpoint.
Why not just use the task's rule checker? For Sudoku a checker exists, and it would dominate any learned signal. That is exactly why we use it as an oracle. The proposal matters for constraints that have no cheap programmatic verifier, which is why we also report results on Even Pixels and polygons. On those tasks, measuring the constraint is either global or approximate.
Decisive Checks
Three checks determine the outcome, and each could fail.
The first is localization. If the BN departure map does not localize rule-checker violations better than chance on Sudoku conflict cells, then guided selection on Sudoku is unjustified, whatever the downstream numbers show. In that case the claim narrows to rendering artifacts on the other benchmarks.
The second is the compute-matched curve. Guided-plus-halting refinement must lie above both the random-selection budget sweep and the self-consistency baseline at every compute level. Compute includes monitor cost, measured in wall-clock time as well as pass counts.
The third is over-revision. We measure the fraction of samples that are valid at some iteration but invalid at the final one. Halting should cut this fraction relative to fixed 50-step refinement. Over-revision is the drift failure that RepoCoder identifies for iterative refinement loops without stopping criteria.
If the first check passes, the BN-statistic ablation decides how strong the final claim is. If BN statistics win, they are a uniquely free signal. If the ablation ties, any feature-density monitor works.
References
- Inference-Time Scaling in Diffusion Models through Iterative Partial Refinement
- Neural Mean Discrepancy for Efficient Out-of-Distribution Detection
- Sample-Free White-Box Out-of-Distribution Detection for Deep Learning
- SAFE: Sensitivity-Aware Features for Out-of-Distribution Object Detection
- BatchNorm-Based Weakly Supervised Video Anomaly Detection
- Effective Restoration of Source Knowledge in Continual Test Time Adaptation
- GraFIQs: Face Image Quality Assessment Using Gradient Magnitudes
- Instance-dependent Early Stopping
- Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision
- BOWLL: A DECEPTIVELY SIMPLE OPEN WORLD LIFELONG LEARNER
- RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation