Recurrent optical flow models estimate how every pixel moves between two video frames. They start from a rough guess and apply the same small correction network over and over, and in current systems every pixel gets the same number of corrections. That is wasteful. A textured patch of static background settles after a pass or two, while an occluded edge or a fast-moving thin object may need every pass available. This note proposes letting the model stop refining a region once that region has settled, so it spends its passes only where the answer is still changing.
The central claim is about training, not the stopping rule. A correction network trained with uniform iteration does not tolerate having its neighbors frozen partway through. Its own uncertainty estimates are also unreliable as a stopping signal early in training. We propose a staged schedule, a curriculum in which the model starts with full uniform refinement, then sees more and more regions randomly frozen, and only then learns to decide for itself where to stop. Under a slowly tightening compute penalty, this schedule should produce a matcher whose accuracy-versus-wall-clock curve dominates the fixed-iteration curve. Without the schedule, the same mechanism should either degrade accuracy or collapse into stopping everywhere.
Where this sits
Iterative refinement is costly in flow because each pass depends on the previous one. RAFT refines a field at 1/8 resolution from a zero initialization. Its loop is a convolutional gated recurrent unit, a small recurrent cell that reads a correlation lookup. It typically runs 32 passes, and it reports 10 FPS on 1088×436 video on a 1080Ti. SEA-RAFT cuts this to 4–12 passes. It does so with a regressed warm start and a Mixture-of-Laplace loss, which predicts a per-pixel spread of the error. It reports 21 FPS at 1080p on an RTX 3090 for its Small variant and at least a 2.3× speed advantage over methods of similar accuracy. Both still spend the budget uniformly across space. That is the gap we target, and SEA-RAFT is the method we build on.
Multi-Scale RAFT takes a different route. It brings back a true coarse-to-fine pyramid running from 1/16 to 1/4 resolution, gaining 14.6% on Sintel Clean. The cost is 13.5M parameters and about 0.3s per frame, again with fixed iterations at every scale. This tells us coarse-to-fine structure buys accuracy. It also shows that the extra scales make uniform compute even more expensive.
The idea of spending refinement only on hard cases has been shown in text reasoning. MAgICoRe sends easy math problems to cheap aggregation and hard ones to iterative refinement. Applying refinement to every problem cost 5.2% on MATH, while selective refinement beat 120-way self-consistency at roughly half the compute. We borrow the motivation but not the method. MAgICoRe routes whole problems using external reward models. We route spatial regions inside a single network using its own uncertainty, and the reward-model overhead MAgICoRe pays would be prohibitive at video rate.
On the curriculum side, two papers supply the reason a staged schedule should matter. Self-Paced Learning shows that admitting harder samples gradually stabilizes non-convex training. Its known failure mode is degenerate pacing that keeps selecting trivial cases, which is exactly the "halt everywhere" collapse we expect from naive joint training. A Task-centric Theory for Iterative Self-Improvement argues that iterative loops amplify errors unless an easy-to-hard ordering stabilizes them. Here the ordering is over how much of the image stops early, rather than over sample difficulty.
Two warnings in the literature constrain the design. SURE documents that matchers give high confidence to gross outliers. Handling Multiple Hypotheses in Coarse-to-Fine Dense Image Matching shows that committing early at high match spread drops accuracy to 51.2%, against 69.2% when several candidate matches are kept. Halting is itself an early commitment, so a region that stops while confidently wrong is the main risk. Our evaluation has to measure it directly.
Core mechanism
We start from the SEA-RAFT architecture. We partition the 1/8-resolution grid into tiles, for example 8×8 cells. Halting decisions are made per tile and not per pixel, because dense convolutions on a GPU do not get faster when individual pixels are masked. At each pass $t$, a light halting head reads three things for each tile: the predicted Laplace scale, the size of the last update, and the change in the correlation lookup. It outputs a stop probability $h_t$.
Tiles that stop are frozen. Their flow and hidden state become fixed context for their neighbors. Active tiles, plus a one-tile border to keep the receptive field correct, are gathered into a compact batch for the next pass. Halting therefore reduces wall-clock time rather than only counting fewer operations.
The training objective adds a compute term to SEA-RAFT's loss:
$$ \mathcal{L} = \mathcal{L}_{\text{MoL}} + \lambda(s)\,\frac{1}{|T|}\sum_{\tau \in T} N_\tau . $$
Here $\mathcal{L}_{\text{MoL}}$ is SEA-RAFT's Mixture-of-Laplace flow loss, $T$ is the set of tiles, $N_\tau$ is the expected number of passes tile $\tau$ receives under the halting probabilities, and $\lambda(s)$ is a compute weight that depends on the training step $s$.
The curriculum has three stages. In the first, $\lambda = 0$ and there is no freezing, which recovers standard SEA-RAFT training. In the second, tiles are frozen at random passes with a gradually increasing probability, biased toward tiles whose error is already low according to ground truth. This teaches the correction operator to work beside frozen neighbors before any learned decision exists. In the third, the halting head takes over and $\lambda(s)$ ramps up gradually rather than switching on all at once.
We also extend the scheme to a two-scale Multi-Scale RAFT variant. Tiles that halt at 1/16 resolution skip their 1/8-resolution passes and are simply upsampled. This is where coarse-to-fine structure enters: the coarse scale decides not only the starting guess but also how much fine-scale work each region receives.
Research Program
The first study is an oracle analysis that bounds what is achievable. On Sintel and Spring, we take a trained SEA-RAFT run with a generous budget and record, per tile, the pass after which its endpoint error stops improving. If most tiles settle within one or two passes and the remainder are concentrated near occlusions, then adaptive halting has real headroom. If error keeps improving everywhere, the idea is weak and we stop there. This study is cheap and comes first.
The second study is the method itself. We train it on SEA-RAFT's published data schedule with identical data, so that the halting head and the staged schedule are the only differences.
The third study is the failure analysis. We measure the error of tiles at the moment they halt and break it down by occlusion and by match spread. The goal is to see whether halting reproduces the confident-but-wrong commitment that BEAMER and SURE describe.
Baselines
The do-nothing reference is SEA-RAFT run at its published iteration counts. It is the strongest efficient recurrent flow method in the file, reporting a 24% EPE reduction and an 18% 1-pixel outlier reduction on Spring over prior published results, and 21 FPS at 1080p for the Small model. Treating it as a single point would flatter us, so we also sweep its uniform iteration count from 1 to 12. This traces a uniform-compute Pareto curve, all on our own GPU. Beating it means lower EPE at matched measured latency across a range of budgets, or matched EPE at measurably lower latency. Fewer average passes alone does not count.
We also include RAFT at 32 passes as the slow but accurate reference, and Multi-Scale RAFT as the reference for coarse-to-fine accuracy. Each is retrained or evaluated on the same data where checkpoints allow.
The ablation that isolates the key ingredient keeps the halting head, the tile gathering and the compute penalty, but trains them jointly from the start with no freezing stage and no ramp on $\lambda$. A second ablation removes only the random-freezing stage.
The file contains no reported per-region adaptive halting result for optical flow (no reported result found on Lacuna for adaptive halting in recurrent flow). The adaptive baselines are therefore ones we construct, and they are described next.
Why not just…?
Why not just threshold SEA-RAFT's Laplace scale at test time, with no retraining? This is the most serious threat. It costs nothing, and SEA-RAFT already predicts a per-pixel spread of the error. We run it as a baseline with the threshold tuned on a held-out split. We expect it to underperform for two reasons. The correction operator was never trained beside frozen neighbors, so active tiles next to frozen ones should drift. And SURE reports that matcher confidence is overconfident on outliers. If the training-free threshold comes within noise of our curve, the curriculum contribution is falsified, and we would report the result as a finding about how cheap halting can be.
Why not just use fewer uniform iterations? SEA-RAFT's own sweep already shows accuracy dropping as passes are removed. The oracle study tells us whether that loss is concentrated in a small set of tiles, and only then can adaptive allocation beat uniform truncation. The uniform-compute Pareto sweep settles this question directly.
Why not just train an adaptive-computation halting head jointly, without any curriculum? That is our main ablation. The prediction, drawn from the self-paced learning failure mode, is collapse toward early halting or an operator that misbehaves near frozen tiles. If joint training matches the staged schedule, the curriculum claim fails, though the halting result would stand.
Why not use a feed-forward matcher like Efficient LoFTR and avoid loops altogether? Efficient LoFTR is fast for semi-dense pose matching, at 56.4 AUC@5° on MegaDepth. But it produces sparse-ish matches, not dense flow, and it cannot revise a coarse commitment. It solves a different problem, so we do not treat it as a direct competitor for dense flow.
Proposed Experiments
We evaluate on Sintel Clean and Final, KITTI-2015 and Spring. Metrics are EPE, 1-pixel outlier rate and KITTI F1-all, alongside measured end-to-end latency at fixed resolution on one GPU model, mean passes per tile, and the fraction of tiles halted with error above 3 pixels.
The decisive check is the Pareto plot of EPE against measured latency on Spring and Sintel Final. Our method must sit below both the uniform SEA-RAFT curve and the test-time-threshold curve. The staged-schedule model must also beat the jointly trained ablation at matched latency.
A secondary check concerns safety. Tiles halted at high match spread, near occlusions, must not show higher final error than the same tiles under full refinement.
Finally, the two-scale variant tests whether halting decisions made at 1/16 resolution can bring Multi-Scale RAFT's accuracy closer to SEA-RAFT's latency. This would make coarse-to-fine refinement affordable rather than a 0.3s-per-frame cost.
References
- RAFT: Recurrent All-Pairs Field Transforms for Optical Flow
- SEA-RAFT: Simple, Efficient, Accurate RAFT for Optical Flow
- Multi-Scale Raft: Combining Hierarchical Concepts for Learning-Based Optical Flow Estimation
- MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning
- Self-Paced Learning for Latent Variable Models
- A Task-centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula
- SURE: Semi-dense Uncertainty-REfined Feature Matching
- Handling Multiple Hypotheses in Coarse-to-Fine Dense Image Matching
- Efficient LoFTR: Semi-Dense Local Feature Matching with Sparse-Like Speed