Spending Refinement Steps Only Where Matching Can Still Help

Modern optical flow networks estimate how every pixel moves between two video frames by refining a guess over and over. Each round looks up how well the current guess matches the second frame and nudges the estimate. Every round costs the same and runs on every pixel. Yet pixels differ in what another round can do for them. Most of a frame is visible, easy surface that settles after a few rounds. A thin band of visible boundary and fine detail keeps improving with more rounds. A third group consists of occluded pixels, which are hidden or newly revealed in the second frame and so have no true match. For that group, the lookup step returns noise no matter how often it is repeated.

We propose a refinement loop that sorts small image tiles into these three groups at every round, using a predicted occlusion map plus the size of the latest update. Settled tiles are frozen. Visible tiles that are still moving get the full lookup-and-update step. Occluded tiles skip the lookup entirely and instead inherit motion from nearby visible tiles that have already settled. The bet is that this cuts wall-clock latency at 1080p without losing accuracy. A second bet is that the occlusion signal matters for more than saving compute: occluded pixels may get worse when they are forced to keep reading meaningless lookups.

Core Claim

In warm-started recurrent flow, we expect most of the remaining iterations to be wasted on two kinds of pixels: those that have already converged, and those that matching cannot improve. A per-tile controller with three choices (freeze, refine with lookup, propagate without lookup) should reach the accuracy of a fixed-iteration model at substantially lower 1080p latency. The decisive ingredient is the third choice. A controller that only knows "converged or not" would keep iterating on occluded tiles, because their estimates oscillate rather than settle. It would therefore waste compute on exactly the pixels where compute cannot help.

Where This Sits in Prior Work

Two lines of work supply the pieces and the motivation.

On the latency side, RAFT introduced the template we build on. It precomputes how well every pixel in frame one matches every pixel in frame two (an "all-pairs correlation volume"), then refines a flow field with a recurrent unit that reads local slices of that volume, typically for 32 or more rounds. SEA-RAFT cuts this to 4–12 rounds by predicting an initial flow directly. It also trains with a two-component Laplace mixture loss whose wide component absorbs occluded and ambiguous pixels. However, it still runs its fixed budget uniformly, and occluded pixels still feed noisy correlation into the update block at every round. DIFT shows that on constrained hardware the bottleneck is memory traffic for the correlation volume. It builds correlation slices just in time instead of storing them, but does so for every location regardless of whether a match exists. GMFlow argues that iteration can be dropped entirely in favor of one global matching pass, though it concedes that sub-pixel precision needs an extra refinement step. IRR shares weights across refinement rounds and estimates occlusion jointly. It explicitly notes that its compute still grows with the number of rounds, because every pixel is processed every time.

These papers supply our base model (SEA-RAFT), our memory strategy (DIFT-style just-in-time correlation, so that skipped tiles also skip correlation construction), and the evidence that uniform scheduling is the remaining inefficiency.

On the occlusion side, PWOC-3D predicts soft occlusion masks and zeroes out matching costs at occluded pixels. OAS-Net reweights the correlation tensor with a learned occlusion map. Both show that keeping unmatchable costs out of the regressor helps, but both are single-pass coarse-to-fine networks. PWOC-3D also constrains our design: zeroing leaves the regressor with no signal at occluded pixels, so we replace the lookup with propagation rather than simply blanking it.

GMA propagates motion into occluded regions through global attention over appearance features inside RAFT. It improves occluded-pixel error, but it adds a full-frame attention to every round and assumes that similar-looking pixels move alike. The authors concede this assumption fails under rotation and camera parallax. We use the same propagation idea but restrict it in two ways. Only occluded tiles receive it. Only nearby visible tiles that have already been frozen act as sources. Both restrictions make it cheap, and the second avoids propagating from estimates that are themselves still wrong.

To our knowledge, no work in this file uses predicted occlusion to decide, per location and per round, what kind of computation to run.

Research Program

The work starts with a diagnostic that can kill the idea before any new architecture is built. We take a pretrained SEA-RAFT model and run it on Sintel and Spring, both of which provide occlusion masks. We record per-round error reduction separately for three pixel groups: visible pixels away from motion boundaries, visible pixels near boundaries, and occluded pixels.

We then intervene at inference on occluded pixels only. In one condition we replace their correlation input with zeros; in another, with the correlation of their nearest visible neighbor. If occluded-pixel error is unchanged or improves, the lookup step is dead weight for them, and the propagate branch is justified. If error worsens noticeably, the recurrent unit is extracting real signal from occluded lookups. In that case we would drop the three-way design and fall back to a two-way freeze/refine controller, which would make this a much smaller contribution.

If the diagnostic supports the idea, the model adds three components to SEA-RAFT.

The first is a lightweight matchability head. It runs on 1/8-resolution features alongside the initial flow regressor. It is supervised with ground-truth occlusion labels from FlyingThings3D, Sintel and Spring. On data without labels, it falls back to the forward-backward consistency heuristic. That heuristic is weak near motion boundaries, as the PWOC-3D discussion notes, so the head is trained mainly on synthetic data.

The second is a per-tile controller, for example over 8×8 tiles of the 1/8-resolution grid. At each round it reads the tile's matchability score, the size of its last update, and its round index, and picks one of the three actions. It is trained with a straight-through relaxation and a compute penalty proportional to the number of tiles that receive lookups. We also distill from an oracle schedule computed with ground-truth occlusion and per-pixel convergence.

The third is the execution path that turns tile decisions into real speed. Active tiles are gathered into a packed batch. Correlation is built just in time only for those tiles, as in DIFT. The update block runs only on the packed batch. Propagated tiles receive a small local attention over frozen visible tiles within a fixed window.

All latency is measured end to end on an RTX 3090-class GPU at 1080p, because skipping work only matters if the gather and scatter overhead is smaller than the saved compute.

Baselines

The naive reference is SEA-RAFT with its iteration count simply lowered, swept from 1 to 12. This traces the accuracy-latency curve that any adaptive scheme must beat. Our method beats it only if it lies strictly above that curve on Spring, not just at a single chosen point.

The strongest existing method on this exact question is SEA-RAFT itself at its standard budget. The file reports that its small variant runs 1080p at 21 frames per second on an RTX 3090 and achieves a 24% endpoint-error reduction on Spring over prior published results. No absolute Spring EPE value for SEA-RAFT appears on Lacuna, so we will re-evaluate the released model ourselves under identical conditions. We will fine-tune it on the same data, for the same number of steps, as our model, so that extra training does not explain any gain. Beating it means matching its overall and unmatched-region Spring endpoint error within noise at no more than about two-thirds of its wall-clock time, or improving unmatched-region error at equal time.

GMA serves as the accuracy reference for occluded pixels. For Sintel Clean, the file reports 1.08 endpoint error for GMFlow against 1.61 for RAFT; GMFlow serves as the single-pass latency reference. Both will be retrained on our data mixture.

Two ablations isolate the key ingredient, and both use the identical tile-packing execution path. The first is a convergence-only controller: a two-way freeze/refine controller driven by update size alone. The second is an uncertainty-driven controller that uses SEA-RAFT's own mixture-of-Laplace weight in place of our matchability head. We also report an oracle that uses ground-truth occlusion, to bound how much the controller leaves on the table.

Why not just…?

Why not just run fewer iterations? SEA-RAFT already shows that a good initial flow makes few iterations viable, so this is a real baseline rather than a straw man. Our naive reference sweep tests it directly. What it cannot do is spend the saved budget on boundary tiles. The Spring paper (Spring) reports outlier rates climbing from about 5.7% overall to over 60% in high-detail regions. That gap suggests uniform budgets fail exactly where per-tile allocation would concentrate effort.

Why not just stop each pixel once its updates become small? This is the most serious threat. It needs no occlusion head and could capture most of the speedup. Our hypothesis is that occluded tiles do not converge cleanly, so a convergence-only rule keeps paying for their useless lookups. The convergence-only ablation, run with identical tile packing, settles this. If it matches our controller on both latency and unmatched-region error, the occlusion contribution reduces to the diagnostic finding alone, and we will report it that way.

Why not just use SEA-RAFT's existing uncertainty as the occlusion signal? That uncertainty is free and already trained. However, its wide Laplace component absorbs motion blur and textureless ambiguity as well as occlusion. Blurred but visible pixels may still benefit from lookups. The uncertainty-driven ablation measures whether this conflation costs accuracy on Spring's high-detail and unmatched masks.

Why not just drop iteration and use single-pass global matching? GMFlow itself leaves sub-pixel refinement open and reports a generalization gap on KITTI. Its global softmax also grows expensive at high resolution, which matters most for Spring at 1080p. We include it on the latency-accuracy plot rather than argue the point.

Decisive Checks

The idea stands or falls on three results.

The first is the inference-time correlation ablation on occluded pixels. If occluded-pixel error rises noticeably when their lookups are removed, the propagate branch is unjustified and the design collapses to two-way gating.

The second is the gap between our controller and the convergence-only controller on Spring. Most of it should appear in unmatched-region error and in the share of lookups spent on occluded tiles.

The third is end-to-end 1080p wall-clock time with gather and scatter overhead included. A reduction in floating-point operations that does not show up as wall-clock speedup would count as failure.

Zero-shot KITTI 2015 evaluation will check that the matchability head, trained mostly on synthetic labels, does not misroute visible pixels in real driving scenes.