Staggered, Damped Updates for Refinement Networks That Can Run Longer at Test Time

Many neural networks produce their answer by repeatedly refining an internal estimate. A shared block is applied again and again: a flow field is corrected, a maze path is extended, a set of object slots is sharpened. In principle, such a network could be given more iterations on harder inputs than it saw in training. In practice, the outcome of extra iterations is uncertain. The estimate may keep improving, stall, drift away from the answer, or blow up.

This note proposes a specific reason for that failure and a specific fix. Each refinement step is a numerical integration step of some underlying continuous process. Most refinement blocks take what amounts to a forward-Euler step: they update everything at once, using the current state. Numerical analysis has long shown that this kind of step quietly adds energy to oscillating systems, so trajectories spiral outward. We propose a refinement block built the way physicists build long-run simulators. The features are split into a "position" half (the current estimate) and a "momentum" half (the pending correction). The two halves are updated in a staggered order, so each half reads the other's freshly updated value. One learned damping coefficient then sets how quickly the process settles.

Core Claim

The claim is narrow. If the only thing that changes is the update order, from simultaneous to staggered with the same parameters and the same damping, a weight-tied refinement network should degrade much more gracefully when run for many more iterations than it was trained with. The settling rate should also become an explicit knob that can be adjusted at test time.

This idea comes from symplectic integration, meaning numerical schemes that preserve the geometric structure of a conservative physical system. The research file describes a key property of such schemes: they exactly conserve a slightly perturbed "shadow" energy. Their trajectories therefore cannot spiral outward or collapse over long horizons. Simultaneous (Euler) updates lack this property.

A purely conservative update would never converge, and refinement needs convergence. For that reason we use the damped variant, often called conformal symplectic, in which phase-space volume shrinks at a fixed, known rate per step. In this design, contraction comes only from the damping term we chose. None of it comes from discretization artifacts.

Prior Work and Its Role

The method comes from Optimization with Momentum: Dynamical, Control-Theoretic, and Symplectic Perspectives. That paper studies momentum methods as discretized second-order ODEs. It shows that naive integration introduces artificial energy growth that makes discrete iterations oscillate or diverge, and that structure-preserving discretizations carry the continuous-time guarantees over to discrete algorithms.

We borrow the diagnosis and the cure, but apply them to a different object. That paper discretizes the optimizer, the procedure that updates the weights. We discretize the forward pass of a learned refinement operator, where the "force" acting on the estimate is itself a trained network rather than the gradient of a fixed loss. As a result, none of its convergence theorems transfer directly. What transfers is the structural argument that update order controls long-horizon energy behavior. Whether that argument survives when the force is learned is precisely what our experiments test.

The research file compiled for this note contains no papers on iterative feature refinement and no search results on whether this combination has been tried. We therefore cannot rule out close prior work, and the novelty of the idea should be treated as unverified.

Research Program

We would build a drop-in refinement cell that works as follows. The state consists of an estimate $q$ and a correction $p$. A learned network $F(q, x)$ plays the role of the force on the estimate, conditioned on the input $x$. Each iteration applies:

$$p_{k+1} = e^{-\gamma h}\, p_k + h\, F(q_k, x), \qquad q_{k+1} = q_k + h\, G(p_{k+1}).$$

Here $h$ is a step size and $\gamma \ge 0$ is a learned scalar damping coefficient. $G$ is a light readout from the correction to the estimate. In this simplest form $G$ is the identity; when $G$ is a learned network, the update keeps the staggering but is no longer guaranteed to be symplectic. The essential feature is that the $q$ update reads $p_{k+1}$, the momentum just computed, rather than $p_k$. That ordering is the symplectic-Euler structure described in the file. A leapfrog variant, which makes symmetric half-steps, would be a second-order alternative.

When $G$ is the identity, phase-space volume contracts by exactly $e^{-\gamma h}$ per step, so the settling rate can be read directly from $\gamma$. We will not claim the same guarantee for a learned, nonlinear $G$. Instead, we will measure the contraction empirically.

The analysis has three parts. The first is a backward-error check: on trained cells, we fit the conserved shadow quantity predicted for the undamped case and measure how far trajectories drift from it. The second is a sweep over test-time step size and iteration count. The third is a test of whether rescaling $\gamma$ at test time trades accuracy against compute in a predictable way. We will train with a fixed number of iterations $K$ and evaluate at $K$, $2K$, $4K$ and $10K$ iterations.

Baselines

The naive reference point is the same shared block used with an ordinary residual update, $q_{k+1} = q_k + h F(q_k, x)$, with no momentum. This represents what an unconstrained refinement network does by default.

The key ablation is the synchronous cell. It has identical parameters, identical damping, and identical $F$ and $G$, but computes $q_{k+1}$ from $p_k$ instead of $p_{k+1}$. Only this comparison isolates the effect of update order. Without it, any gain could simply come from adding momentum or damping.

The strongest practical competitor is a gated recurrent refinement cell: a GRU-style update, in which learned gates decide how much of the old state to keep. This is the design most refinement systems actually use. Its gates can learn contraction on their own, so it is the real threat to our proposal.

The fixed-point alternative solves for the refinement's equilibrium directly with a root-finder rather than unrolling iterations. It addresses the same goal of being able to "run until done" by a different route.

All methods will use matched parameter counts, the same training iteration count $K$, the same data, and an equal hyperparameter search budget. That budget will include a search over damping and gate initializations for the competitors, not only for our method.

To count as beating a baseline, our cell must show a smaller drop in accuracy from $K$ to $10K$ iterations than both the gated and synchronous cells, while matching their accuracy at $K$ within noise.

For reported numbers, there is no reported result found on Lacuna for any of these baselines on any benchmark. All baseline figures will be produced by our own runs.

Why not just…?

Why not just use a gated recurrent cell, which already learns when to stop updating?

This is the most serious alternative, and it could match our results. Gates can learn to shrink the update toward zero, which also stops drift. Our bet is that gates learn contraction only on the iteration counts seen in training, so their behavior beyond $K$ is not controlled. A structural guarantee, by contrast, holds at any iteration count. The head-to-head extrapolation curve settles this question. If the gated cell's degradation from $K$ to $10K$ is as flat as ours, then the remaining contribution is interpretability: a single damping knob instead of opaque gates. We would report the result that way.

Why not just train with a random number of iterations so the network learns to be stable?

Randomized-depth training is cheap and plausibly effective within the range of counts it samples. It is also orthogonal to our proposal and can be combined with it. We will run every cell, including ours, with and without randomized iteration counts. If staggered updates add nothing once random-depth training is used, the claim fails.

Why not just solve for the fixed point directly?

Root-finding converges only when a fixed point exists and the solver is well-conditioned. It also gives up the anytime property, meaning a usable intermediate answer after any number of steps. We include it as a baseline and report both accuracy and failures to converge.

Proposed Experiments

The research file supplies no benchmarks, so the choices below are our own and are not drawn from cited results.

The first setting is synthetic algorithmic extrapolation. A weight-tied network learns maze solving or prefix sums on small instances and is tested on larger ones that require more iterations. These tasks directly measure whether extra test-time iterations help.

The second setting is iterative dense prediction: optical-flow-style refinement on a standard flow benchmark, measured by endpoint error as a function of iteration count.

The decisive check is the synchronous-versus-staggered ablation at $10K$ iterations. Supporting checks are the measured drift of the shadow quantity and the predictability of the accuracy–compute trade-off when $\gamma$ is rescaled at test time.

A null result would also be informative. If update order does not matter once the force is learned, then the energy-drift explanation for refinement failure is wrong, and the effort should go into training procedures instead.