Scratch Dimensions for Faster Latent Refinement

Many fast text generators avoid writing one token at a time. They draft a whole output in parallel and then improve it over several passes. Each pass costs time, so the number of passes sets the latency. Iterative Refinement in the Continuous Space for Non-Autoregressive Neural Machine Translation showed that these passes can be cheap if the model edits a tiny continuous code instead of the words themselves. It uses an 8-number vector per position and nudges it along a learned gradient. The words are decoded only once, at the end. That design reached a 6.2x speedup over an autoregressive Transformer on WMT'14 En→De, losing 0.9 BLEU. The 8-number code has a cost, though. A refinement step that sees only the current code has no memory of how it got there, so every path through the same point must continue the same way. Paths that need to pass through each other, or reorder, have to take many small detours.

The proposal is to give the refiner a handful of "scratch" dimensions. These extra numbers start at zero, are updated alongside the code, and are never read by the decoder. Decoding cost stays the same, and the per-step cost grows only slightly. The refiner gains working memory that lets refinement paths cross in the visible code. The question is whether this reaches a given translation quality in fewer passes, and therefore at lower latency, than the unpadded refiner and its obvious cheaper alternatives.

Core Claim

Zero-padding the state of a continuous refinement process should reduce the number of steps needed to converge, at a fixed decoder. The padding idea comes from Augmented Neural ODEs, which pads the state of a neural ordinary differential equation (a network whose hidden state evolves continuously under a learned velocity field) with zeros. That paper showed that a flow without padding cannot let trajectories cross. When the target mapping requires crossing, the learned field becomes jagged, and the solver's function-evaluation count drifts upward. Padding kept that count low and sped training roughly 10x on MNIST. Approximation Capabilities of Neural ODEs and Invertible Residual Networks proved that padding with as many zeros as there are input dimensions removes the topological obstruction entirely.

The setting here is different from both papers. They studied the forward map of a classifier or density model. This proposal applies padding to the inference-time refinement loop of a sequence generator and measures steps-to-quality and wall-clock latency, not training speed.

The claim is falsifiable. If the scratch dimensions help, BLEU should rise faster per refinement step. The benefit should be concentrated on sentences whose initial draft needs large reorderings. It should also disappear if the scratch state is given the same training procedure but zero width.

The strongest direct evidence that this transfers comes from speech. Memory-Efficient Multi-Step Speech Enhancement with Neural ODE found that multi-step denoising beat single-step processing (up to +0.16 PESQ) only when dimension augmentation was present; without it, gains were marginal. That paper never measured inference latency or steps-to-quality. It established that augmentation is what makes multiple steps worth taking, not that it makes fewer steps sufficient. This work tests the second claim.

Research Program

The base system is the score-function refiner of Iterative Refinement in the Continuous Space for Non-Autoregressive Neural Machine Translation. Each target position has a latent $z \in \mathbb{R}^d$ with $d = 8$, and a network predicts the direction that increases $\log p(y \mid x, z)$. To this we attach an auxiliary state $a \in \mathbb{R}^p$, initialized to zero, and update both together:

$$[z, a]_{k+1} = [z, a]_k + \eta \, g_\theta([z, a]_k, x),$$

Here $k$ indexes refinement steps, $\eta$ is the step size, $x$ is the source sentence, and $g_\theta$ is the refinement network. The decoder reads only $z_K$ after the final step $K$.

The scratch state $a$ has no gradient target of its own, so the original paper's regression objective cannot train it. We instead unroll $K_{\text{train}}$ steps and backpropagate the final decoding loss, summed over a few intermediate steps so the model is rewarded for being good early. This change in training procedure is itself a confound, which is why the key ablation below keeps it fixed and removes only the padding.

Three questions organize the work.

The first is the width sweep: how steps-to-quality depends on $p$, from 0 to $2d$. We expect diminishing returns, and possibly harm at large $p$, because Augmented Neural ODEs reports degraded generalization with excessive padding.

The second is whether the scratch state can be dropped late in refinement to recover per-step cost, the "asymmetric unpadding" gap. We test this by freezing $a$ after step $k^*$ and measuring the quality loss.

The third is what the scratch state actually encodes. We probe whether $a$ predicts the eventual edit distance or reordering between draft and output. We also count "near-crossings": pairs of trajectories whose $z$ values come within a small distance and then diverge. A refiner without padding should produce few of these, and a padded one many.

Two Sides of The Same Coin: Bridging Deep Equilibrium Models and Neural ODEs via Homotopy Continuation supplies part of the method and part of the competition. It shows that padded dimensions act as the search space for an equilibrium point. It also reports a 2.5x inference speedup on CIFAR-10 from learning a shared initial point. We therefore also try a learned, input-dependent initialization of $a$ in place of zeros, and keep the shared-initialization trick as a separate baseline.

A second testbed checks that the effect is not specific to translation. We use the Voice Bank + DEMAND setup from Memory-Efficient Multi-Step Speech Enhancement with Neural ODE, where augmentation is already in place. There we measure PESQ against solver function evaluations and wall-clock time, with and without augmentation, which that paper did not report.

Baselines

The naive reference point is the original unpadded score-function refiner from Iterative Refinement in the Continuous Space for Non-Autoregressive Neural Machine Translation, retrained in our codebase with the same distilled data and encoder. It reports a 6.2x speedup over the autoregressive model on WMT'14 En→De at a cost of 0.9 BLEU. The file does not give its absolute BLEU, so we report our reproduction alongside that published ratio.

The strongest existing method on the axis of quality per pass is the discrete refinement decoder of Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement. It reports 21.54 BLEU with adaptive refinement against a 23.77 autoregressive baseline on WMT'14 En-De, and its code is available at nyu-dl/dl4mt-nonauto. It is run on the same distilled data and hardware, with its Jaccard-based stopping rule.

Beating the baselines means two things. The padded refiner must reach the unpadded refiner's converged BLEU in measurably fewer steps, with lower median and 90th-percentile latency at batch size 1. Its BLEU must also be no worse than that of the unpadded refiner at every step budget from 1 to 10.

The ablation that isolates the key ingredient is the unpadded refiner ($p = 0$) trained with exactly the same unrolled, multi-step objective. If this matches the padded model, the gain came from the training change, not from the scratch dimensions.

All variants receive the same hyperparameter-search budget and the same number of training updates. Each variant also has its parameter count matched by widening $g_\theta$. Every result is averaged over three seeds.

Why not just…?

Why not just make the visible latent bigger, say 16 dimensions instead of 8 plus 8 scratch? This is the most serious threat. A larger visible latent also gives trajectories room to move. The difference is that the decoder must read and be trained on every visible dimension, while scratch dimensions impose no constraint on what the decoder sees. A larger $z$ may make the latent harder to model, and the original paper deliberately kept $z$ small. We include visible widths $d \in \{8, 12, 16\}$ at matched total state size. If $d = 16$ matches $d = 8, p = 8$ on steps-to-quality, the "scratch" framing is wrong, and we will report it as "a larger latent helps."

Why not just add momentum, which also carries memory across steps? Heavy-ball momentum is a fixed, hand-designed auxiliary state of width $d$, so it is exactly a special case of this proposal. If learned scratch dimensions do no better than momentum with a tuned coefficient, the learning adds nothing. We run momentum and Adam-style refinement steps over the same $z$ as baselines. We expect learned scratch state to win specifically on sentences that need large reorderings, where the right "memory" is not just a running average of past gradients.

Why not just distill the refinement loop into one step? DCFold: Efficient Protein Structure Generation with Single Forward Pass collapsed AlphaFold3's iterations into one pass for a 15x speedup. Its Posebusters V2 success rate fell from 82.9% to 78.6%, and the authors note mild concentration of the output distribution. That evidence comes from proteins, not translation, so it does not settle the question. We distill the unpadded refiner into a one-step predictor and place it on the same latency-versus-BLEU plot. If it dominates, the honest conclusion is that steps should be removed rather than accelerated.

Why not just learn a better starting point? Two Sides of The Same Coin: Bridging Deep Equilibrium Models and Neural ODEs via Homotopy Continuation obtained a 2.5x inference speedup by learning a shared initial state. A better start shortens every path but cannot let paths cross. We run a learned shared initialization of $z$ with $p = 0$, and test whether combining it with padding yields additive gains.

Proposed Experiments

On WMT'14 En→De, the primary benchmark, the main result is a BLEU-versus-steps curve and a BLEU-versus-latency curve for every variant above. They are measured at batch size 1 on both GPU and CPU, since Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement found CPU and GPU speedups differ (up to 5x versus 2–3x). We also report token-repetition rates, since continuous refinement was previously shown to reduce repetition. WMT'16 Ro→En and IWSLT'16 De→En serve as replication sets. The file reports no numbers for them, so our baselines there are our own reproductions.

The first decisive check is the matched-objective ablation. The padded refiner should reach the $p = 0$ refiner's step-10 BLEU by step 5 or earlier. If it reaches that BLEU only at step 9 or 10, the idea yields no meaningful latency gain.

The second decisive check is stratification. We bucket test sentences by the reordering distance between the first draft and the final output. The padded model's advantage should grow with reordering distance. If the advantage is flat across buckets, the topological explanation is wrong even if average gains exist.

The third decisive check is the near-crossing count. Padded models should show more trajectory pairs that pass close in $z$ and then diverge.

On Voice Bank + DEMAND, we plot PESQ and STOI against solver function evaluations and wall-clock time for augmented and unaugmented ODE enhancers. The paper there reports only gains over single-step processing (+0.16 PESQ, +0.5% STOI) and no latency, so any steps-to-quality improvement there would be a new, independent confirmation.