Optical flow means estimating, for every pixel, where it moves between two video frames. Flow networks are trained almost entirely on rendered data, because real video has no free per-pixel motion labels. They then fail in a characteristic place: the outlines of moving objects. There, the motion of a foreground object "bleeds" into the background, or the estimate breaks off along a shadow or a painted road marking.
Our hypothesis is that part of this failure is a learned shortcut. In synthetic training sets, object outlines almost always coincide with strong, clean intensity edges, so a network can learn to "stop smoothing the motion where the image has an edge." Real cameras break that correlation. Low-contrast object outlines go unrecognized, and high-contrast texture creates false cuts. We propose to replace the image-edge cue with a cue that transfers across the gap. Two pixels belong to the same moving surface if their trajectories agree over several frames. We would build this into a flow transformer by factoring attention into separate spatial and temporal steps. The temporal step computes how well neighboring pixels' trajectories agree across frames. That agreement then decides which pixels the spatial step is allowed to mix.
Core Claim
The claim has two parts, and both can be falsified. The first is diagnostic. Current flow networks trained on synthetic data rely on intensity edges to place motion boundaries, and this reliance accounts for a measurable share of their boundary error on real data. The second is constructive. A spatial attention whose weights are biased by multi-frame trajectory agreement, rather than by appearance similarity, recovers sharp boundaries without that shortcut. It should therefore narrow the gap between boundary accuracy on synthetic and on real data.
Neither part follows automatically from existing work. Appearance-guided attention, as in GMA, routes motion between pixels that look alike. GMA's own stated failure is camouflage and articulation, which are exactly the cases where appearance and motion disagree. The motion-affinity bias we propose can be written as
$$a_{ij} \propto \exp\!\left(\frac{q_i^\top k_j}{\sqrt{d}} - \lambda\, \frac{1}{T}\sum_{t=1}^{T} \lVert u_i^{t} - u_j^{t} \rVert\right),$$
where $a_{ij}$ is how much pixel $i$ draws on pixel $j$ within a local window, $q_i, k_j$ are the usual query and key features of dimension $d$, $u_i^t$ is the current flow estimate of pixel $i$ toward frame $t$ among $T$ neighboring frames, and $\lambda$ is a learned, per-head strength. The term is small inside a rigidly moving region, where trajectories agree over time. It becomes large across a true motion boundary, where trajectories diverge. It stays small across a painted edge, which moves with its surface.
Relation to Prior Work
Several papers motivate the proposal by exposing the gap. Classical boundary-preserving methods tie motion discontinuities to image edges. EpicFlow interpolates sparse matches with an edge-weighted geodesic distance. From Pixels to Layers argues that static color and edge cues predict motion boundaries. These methods work, but the file records their known failure: static edges do not distinguish texture from motion discontinuity. Our proposal keeps their goal, smoothing that stops at true boundaries, but replaces the edge detector with temporal evidence. ABDA-Flow also seeks a boundary cue that survives a domain shift, here day to night. It obtains that cue from event cameras and cuts Dark-DSEC end-point error to 0.74. We pursue the same idea using only ordinary RGB frames over time, which also makes it applicable to legacy synthetic data.
Two papers supply the method. WAFT drops the low-resolution cost volume (a table of match scores between all pixel pairs) and instead warps features at half resolution with a transformer update module. It reports first place on Sintel, Spring and KITTI at 9.2 GiB of memory. Its outputs are sharp, but it works on two frames only and fails when the initial flow is far off. We build on WAFT's high-resolution update module, using its public code (princeton-vl/WAFT). Our addition is decoupled spatio-temporal attention: spatial attention within a frame, and separate temporal attention across frames along each pixel's warped trajectory. This costs $O(TN^2 + NT^2)$ for $T$ frames of $N$ tokens rather than $O(T^2N^2)$. With local spatial windows, half-resolution operation stays affordable.
Other papers constrain the design. Sparse Video-Gen shows that attention heads naturally split into spatial and temporal roles. It also shows that forcing decoupling too early irreversibly degrades output. Our version of that warning is circularity: the motion bias depends on the network's own flow estimate, which is poor at the first refinement iterations. We therefore set $\lambda$ to zero at the first update step and let the network learn how quickly to increase it. TS2-Net argues for delaying temporal mixing to protect fine spatial features, which points the same way. Yang et al. report that joint space-time attention lets strong static spatial correlations drown out weak temporal signals. That is the reason we keep the temporal trajectory signal in its own branch rather than folding it into one large attention.
Finally, the cost-volume transformers that factor attention differ in what they factor. FlowFormer splits a two-frame cost volume into attention within each pixel's match map and attention across pixels. That split runs over space and displacement, not over time, and it has no boundary prior. FlowFormer++ pretrains that encoder on real YouTube-VOS video by masked reconstruction, with masks that ignore object boundaries. CRAFT sharpens boundaries through learned cross-frame attention, but on two frames and with no domain-adaptation mechanism. In all three, any boundary improvement comes as a by-product. None of them tests whether the network is relying on edges.
Research Program
The first stage is a shortcut audit. We would build an edge–motion decorrelation probe from Sintel and FlyingThings3D sequences, exploiting the fact that ground-truth flow lets us edit images without changing labels. One variant pastes high-contrast textures that move with the underlying surface, so it adds edges but no motion boundaries. A second variant blends foreground and background color statistics near true occlusion outlines, so the boundaries remain but lose contrast. We would measure how much off-the-shelf WAFT, FlowFormer++ and GMA degrade at boundaries under each edit. We would then check whether the same pattern appears on real KITTI-2015. To do that, we split KITTI's foreground boundary pixels by local image-gradient strength and compare error on low-contrast and high-contrast object outlines. If synthetic-trained models are not edge-dependent, the probe will show it and the remaining program changes direction. This stage is cheap and comes first for that reason.
The second stage builds the model. Starting from WAFT, we feed $T=4$ frames centered on the target pair. A temporal attention block lets each pixel attend to features sampled along its own warped trajectory, while the spatial attention blocks in the update module receive the trajectory-agreement bias above. Training follows the standard path: FlyingChairs, then FlyingThings3D, whose sequences provide multiple frames. An edge-decorrelation augmentation, built the same way as the probe, removes the shortcut from training data so that the model must rely on something else.
The third stage is optional real-video adaptation on unlabeled YouTube-VOS. It would use photometric and multi-frame forward–backward consistency losses and would update only the temporal branch. This tests whether the trajectory cue can be refined on real data in a way that the edge cue cannot.
Baselines
The naive reference point is WAFT run exactly as released: two frames, trained on the same synthetic data with no augmentation changes. It measures how much the whole program adds over simply using the strongest available two-frame warping model.
The strongest existing competitor is again WAFT, retrained under our exact data schedule, iteration count and resolution. Two facts make it the right target. It is the best-ranked method in the file on Sintel, Spring and KITTI, and it already addresses boundary blur by working at high resolution. Any gain we show must therefore come from the temporal cue, not from resolution. We found no reported WAFT end-point error value on Lacuna, only rankings and a "more than 31%" KITTI zero-shot error reduction over earlier warping methods. We will therefore reproduce its numbers from the official code before comparing.
As the strongest cost-volume competitor, we include FlowFormer++, retrained or evaluated from released weights under the same split. Its published results are Sintel clean 1.07 and final 1.94 end-point error, and KITTI-2015 test F1-all of 4.52 (FlowFormer++). CRAFT's KITTI foreground outlier ratio of 5.85% (CRAFT) is the reported reference for boundary-heavy real-world error. All retrained models get the same GPU budget and the same hyperparameter-search budget.
Beating these baselines means two things on zero-shot transfer from synthetic training. First, lower error in Sintel's near-boundary band (pixels within 10 of a motion boundary, the $d_{0-10}$ split) and lower KITTI Fl-fg than retrained WAFT, with no loss in overall end-point error. Second, a smaller degradation on the probe's low-contrast and fake-edge variants.
Several ablations isolate the key ingredient. In the first, the temporal attention branch is kept but $\lambda$ is fixed at zero, which separates the value of extra frames from the value of the gating. In the second, the motion-agreement term is replaced by an appearance-affinity term in the style of GMA, which separates motion-based from appearance-based gating. In the third, the bias is computed from two frames only ($T=2$), which tests whether temporal agreement matters beyond a single flow field. In the fourth, the augmentation is applied alone to the unmodified WAFT.
Why not just…?
Why not just augment the synthetic data to break the edge shortcut? This is the most serious threat, and we incorporate it rather than argue around it. Augmentation can stop a network from trusting edges. It does not give the network anything else to use at a two-frame occlusion boundary, where the occluded side has no match by construction. Our prediction is that augmentation alone makes WAFT more robust on the fake-edge probe but blurrier at true boundaries. The "augmentation-only" ablation tests this directly. If augmented two-frame WAFT matches our full model on $d_{0-10}$ and Fl-fg, the gating is unnecessary and we would report that.
Why not just give WAFT more frames, by warm-starting from the previous flow or adding temporal attention without the bias? More frames help with large displacements, which is WAFT's documented failure. That does not by itself stop smoothing from crossing boundaries. The $\lambda=0$ ablation answers this question exactly. If it captures most of the gain, the contribution reduces to "multi-frame WAFT," which is useful but a different and smaller claim.
Why not just post-process with edge-aware filtering, or use GMA-style appearance attention? Both put the edge or appearance cue back in, and the shortcut audit targets that cue. EpicFlow-style filtering also takes 16.4 seconds per frame on a CPU (EpicFlow). We will still apply a cross-bilateral filter to WAFT outputs as a cheap check, and the appearance-gating ablation stands in for GMA. On the low-contrast probe, both should do worse than motion gating. If they do not, the motivating hypothesis is wrong.
Why not just use full joint space-time attention? At half resolution its cost of $O(T^2N^2)$ is prohibitive. The file also records evidence that joint attention lets static spatial structure drown out temporal signals (Yang et al.). We will compare against joint attention at matched compute, which forces it to run at lower resolution. If it still wins on boundary error, decoupling is not justified.
Proposed Experiments
The decisive experiments are three. The first is the shortcut audit on the decorrelation probe and on contrast-stratified KITTI-2015 boundary pixels. It must show edge dependence before anything else is worth building. The second is a zero-shot evaluation, training on FlyingChairs and FlyingThings3D only, on Sintel clean and final (including the $d_{0-10}$ and occluded-pixel splits), on Spring, and on KITTI-2015 (overall Fl-all and the foreground-object Fl-fg, using neighboring frames released with the benchmark where available). Here the central quantity is the gap between near-boundary error on Sintel and foreground outlier rate on KITTI, compared with the same gap for WAFT. Dark-KITTI2015, where ABDA-Flow reports 3.47 end-point error using event-camera supervision, serves as a stress test: low light flattens image edges, but temporal agreement should still hold. The third experiment adds the optional YouTube-VOS adaptation, to see whether updating only the temporal branch on real video further narrows the boundary gap.
We will also report a learned-$\lambda$ trace across refinement iterations. It shows whether the network chooses to delay motion gating, as the early-decoupling results of Sparse Video-Gen would predict. It also indicates whether the circularity of using the network's own estimates is a real problem. Memory and runtime are reported against WAFT's 9.2 GiB figure, so that any accuracy gain is weighed against its cost.
References
- Learning to Estimate Hidden Motions with Global Motion Aggregation
- EpicFlow: Edge-preserving interpolation of correspondences for optical flow
- From Pixels to Layers: Joint Motion Estimation and Segmentation
- Exploring the Common Appearance-Boundary Adaptation for Nighttime Optical Flow
- WAFT: Warping-Alone Field Transforms for Optical Flow
- Sparse Video-Gen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity
- TS2-Net: Token Shift and Selection Transformer for Text-Video Retrieval
- Neural Combinatorial Optimization for Time-Dependent Traveling Salesman Problem
- FlowFormer: A Transformer Architecture for Optical Flow
- FlowFormer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimation
- CRAFT: Cross-Attentional Flow Transformer for Robust Optical Flow