When a pixel in one video frame is hidden in the next, there is nothing to match it against. Optical flow networks, which estimate where each pixel moves between frames, must then guess its motion from the rest of the image. The most influential fix lets an occluded pixel look across the whole image, find visible pixels that look like it, and copy a weighted average of their motion vectors. That works when the object slides. It breaks when the object rotates, scales, or moves toward the camera. In those cases two visible pixels on the same object have different motion vectors, and their average is not the motion of a third, hidden pixel elsewhere on the object.
We propose to keep the global "look across the image" step but change what is gathered. Instead of averaging motion vectors, each occluded pixel fits a small local motion model (an affine map: translation plus rotation, scale and shear) to the visible pixels it attends to. It then evaluates that model at its own position. The fit needs only a few weighted sums, so it costs the same as the existing averaging step. In the special case where the motion model is pure translation, it reduces exactly to the existing method. The claim is narrow and testable: on hidden pixels whose true neighborhood motion is non-translational, fitting beats copying, and the gap grows with how strongly the motion departs from translation.
Where the two directions meet, and what is still open
The occlusion literature has many ways to stop occluded pixels from damaging training. Forward-backward consistency masks in GeoNet and Depth From Videos in the Wild zero out the loss there. SelFlow distills flow onto synthetic occlusions. SEA-RAFT gives occluded pixels a high-variance mixture component. These methods motivate the problem but do not say what the motion inside the hole should be.
The receptive-field literature supplies the means to reach outside the hole. Methods differ in what they assume about the motion they reach for:
- GMA computes attention between reference-frame appearance features. It aggregates projected local motion features through that attention and feeds the result to RAFT's recurrent updater. The output at a query pixel is a convex combination of other pixels' motion values.
- SKFlow argues against attention and widens the receptive field with 15×15 depth-wise kernels. Its gains saturate and then degrade at 31×31, and its reach remains spatially bounded.
- FlowFormer routes global evidence through transformer layers over compressed cost tokens. It does so at 18.2M parameters.
- OccInpFlow copies motion from appearance-matched regions in a separate, multi-stage inpainting network.
GMA and OccInpFlow share a copying assumption: "looks alike, so moves alike." The research file names this as GMA's main open limitation under articulation and out-of-plane rotation. SKFlow and FlowFormer make no explicit assumption and leave the network to learn one.
The structural prior we need comes from the 3D side. RigidFlow fits one rigid transform per geometric supervoxel with an iterative closest-point procedure, then uses it as a pseudo-label. That is parametric fitting, but over fixed hand-built segments and outside the network. Our proposal moves the fit inside the network and makes the support region soft and global, chosen by learned attention rather than by over-segmentation.
Core Claim
Global attention can return a motion model rather than a motion vector. Let $x_i$ be the coordinate of query pixel $i$ and $f_j$ the current flow estimate at pixel $j$. Let $a_{ij}$ be the appearance attention weight times a predicted visibility confidence for $j$, and $p_j = x_j - x_i$ the relative position. We solve a weighted least-squares affine fit:
$$ \theta_i = \Big(\sum_j a_{ij}\,\tilde p_j \tilde p_j^{\top} + \lambda I\Big)^{-1} \sum_j a_{ij}\,\tilde p_j f_j^{\top}, $$
where $\tilde p_j = [p_j; 1]$ is the homogeneous relative coordinate, $\lambda$ a small ridge term, and $\theta_i$ a 3×2 matrix. Because $p=0$ at the query, the predicted flow at $i$ is the last row of $\theta_i$, the intercept. If the linear part is forced to zero, this intercept is exactly the attention-weighted mean of $f_j$, which is GMA's operation applied to flows.
The sums expand into attention-weighted moments of absolute coordinates and flows, for example $\sum_j a_{ij} x_j$ and $\sum_j a_{ij} x_j f_j^{\top}$, shifted by $x_i$ afterward. So the whole fit is one attention pass with about a dozen value channels per head, followed by a 3×3 solve per pixel. Multiple heads give multiple motion hypotheses. Each head's weighted fit residual is passed to the recurrent updater as a confidence signal, so the network can fall back to local evidence when no coherent affine support exists.
Research Program
We build the module on the RAFT backbone exactly as GMA does. The only change is replacing the GMA aggregation with the moment-and-solve step, keeping the same attention keys, the same 1/8 resolution, and the same updater input slot. We run it inside each recurrent iteration on the current flow estimate, so the fit improves as visible-pixel flow improves.
The analysis has three parts:
- A stratified error measure. We split Sintel's unmatched pixels by how non-translational their ground-truth motion is. The score is the residual of a translation-only fit to ground-truth flow in a fixed window around the pixel, relative to the residual of an affine fit. This gives a curve of occluded-pixel error against motion complexity.
- A whole-object occlusion slice. We isolate pixels where no same-object visible pixel exists. Here we expect neither copying nor fitting to help. This slice delimits the claim rather than testing it.
- Cost. We measure memory, MACs and wall-clock time against GMA. SKFlow's experience is a warning that theoretical FLOPs and real throughput diverge.
Baselines
The do-nothing reference is RAFT RAFT with local correlation and 3×3 updater convolutions, retrained under our schedule. The file reports that RAFT reduces Sintel Final error by 30% relative to earlier leaders, but gives no raw Sintel unmatched figure. We therefore regenerate all numbers ourselves.
The strongest direct competitor is GMA, because it is the method our module generalizes. It is the canonical attention-based occlusion fix on the same backbone. The file reports a 13.6% Sintel Final average end-point error improvement over RAFT, but no reported raw EPE-unmatched number was found on Lacuna for GMA.
SKFlow is the strongest non-attention competitor. The file reports a 7.92% reduction in Sintel Final unmatched error relative to GMA, which makes it the bar to clear on occluded pixels. FlowFormer, at 2.183 Sintel Final average end-point error, is a reference for what a much larger model achieves. It is not a like-for-like comparison.
All RAFT-family models are trained from scratch with identical data (FlyingChairs, then FlyingThings3D, then the standard Sintel/KITTI/HD1K mix), identical iteration counts, batch size, optimizer, and an equal small hyperparameter search per method. We use three seeds each.
Two ablations isolate the key ingredient:
- GMA-on-flows. Our module with the affine part zeroed, that is, attention-weighted averaging of current flow estimates. This separates "aggregate flows" from "fit a model."
- GMA plus coordinate values. GMA whose value vectors are augmented with the attended pixels' coordinates and coordinate-flow products. This lets the updater learn the extrapolation implicitly rather than via a closed-form solve.
Beating the competitors means three things:
- Lower Sintel Final EPE-unmatched than both GMA and SKFlow across seeds.
- A gap over GMA that widens monotonically with the non-translational score.
- No loss on matched pixels or KITTI 2015 F1-all, at MAC cost within a few percent of GMA.
Why not just…?
Why not just use larger kernels, as SKFlow does? A 15×15 kernel at 1/8 resolution reaches about 120 input pixels. Larger kernels degraded in SKFlow's own study, so occlusions wider than that horizon stay unreachable. Our stratified evaluation will additionally bin unmatched pixels by distance to the nearest visible same-object pixel. We expect SKFlow's advantage to vanish beyond its kernel reach while attention-based methods do not. If SKFlow matches us even in the far, rotating bins, the fitting claim fails.
Why not just give GMA positional information and let the updater learn the extrapolation? This is the real threat. The weighted moments we use are exactly what such augmented values would carry, so a sufficiently trained updater could in principle learn the solve. The GMA-plus-coordinate-values ablation settles it. If it matches our module, the honest conclusion is that "values must carry position" is the finding, and the closed-form solve is merely a convenience. We would report it that way. We expect the explicit solve to help mainly where training data has few rotating occluded examples, because a learned updater must discover a ratio of moments that the solve computes directly.
Why not just add frames, as M2Flow does? Multi-frame models recover hidden motion when the pixel was visible in an earlier frame. M2Flow reports 25.29 Sintel Final EPE-occ in the unsupervised setting. However, multi-frame models pay for four-frame inputs, and their per-frame spatial reasoning stays local. Our module is a two-frame spatial operation and is complementary to them. We treat temporal context as orthogonal and test it only as an optional addition, not as a substitute.
Why not just a bigger transformer like FlowFormer? It may well do better on average. However, it does so with 3.4× RAFT's parameters and extra pretraining (FlowFormer++). Our question is whether a structural change to one cheap module fixes a specific failure. The stratified curve, not the leaderboard average, answers that question.
Decisive Checks
The first check is a controlled diagnostic. We build FlyingChairs-style pairs in which a foreground object rotates about a known center while a second object occludes part of it, and we sweep rotation magnitude. With translation only, GMA-on-flows and our module should tie. As rotation grows, GMA's error on hidden pixels should grow roughly linearly with distance from the visible support times angular velocity, because that is the flow difference an average ignores. Our error should stay near the visible-pixel error. Observing this split, then reproducing the widening gap on Sintel's real unmatched pixels and in KITTI 2015 occluded outliers, would support the claim. A flat gap would refute it.
The second check is cost. We measure memory, MACs and wall-clock time on 1/8-resolution Sintel inputs and require the overhead relative to GMA to be small. The whole-object occlusion slice is reported separately as the region where we predict no improvement.
References
- GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose
- Depth From Videos in the Wild: Unsupervised Monocular Depth Learning From Unknown Cameras
- SelFlow: Self-Supervised Learning of Optical Flow
- SEA-RAFT: Simple, Efficient, Accurate RAFT for Optical Flow
- Learning to Estimate Hidden Motions with Global Motion Aggregation
- SKFlow: Learning Optical Flow with Super Kernels
- FlowFormer: A Transformer Architecture for Optical Flow
- OccInpFlow: Occlusion-Inpainting Optical Flow Estimation by Unsupervised Learning
- RigidFlow: Self-Supervised Scene Flow Learning on Point Clouds by Local Rigidity Prior
- RAFT: Recurrent All-Pairs Field Transforms for Optical Flow
- M2Flow: A Motion Information Fusion Framework for Enhanced Unsupervised Optical Flow Estimation in Autonomous Driving
- FlowFormer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimation