Optical flow is the per-pixel motion between two video frames. Without labeled motion, the usual way to train a flow network is to warp one frame onto the other with the predicted motion and penalize the pixel differences. That signal is wrong wherever a pixel is hidden in the other frame, so methods run the network in both directions and check whether a round trip (forward, then backward) returns to its starting point. Pixels whose round trip fails are treated as occluded and removed from the loss. Modern multi-scale networks also attach a loss to every intermediate resolution, from 1/16 up to full size. This practice, called deep supervision, gives the coarse levels their own learning signal for large motions.
These two ideas collide at the coarse levels. A failed round trip at 1/16 resolution has two possible causes: the pixel really is hidden, or the coarse level simply got the motion wrong. Standard practice masks both cases away, so coarse levels stop learning exactly on the large, hard motions they exist to solve. Our claim is that the scale hierarchy separates the two cases for almost no extra cost. The finest, most refined forward and backward estimates are the best available judge of true occlusion. A pixel that fails the round trip at a coarse level but passes it at the finest level is a coarse-level mistake, not an occlusion. That pixel should get more weight in the coarse loss, not be discarded.
Core Claim
In unsupervised multi-scale flow training, each level's loss should be gated by the occlusion estimate from the finest level, pooled down with area-fraction weights and blocked from gradients. Each level should also receive an extra weight on pixels where its own native round-trip check disagrees with that fine-level verdict. This cross-scale disagreement is a label-free signal of "matchable but mis-estimated." It is cleaner than the photometric-error signal used for hard-pixel weighting in prior work, because photometric error in real footage is confounded by lighting changes and texture.
The mechanism needs a little notation. Let $M$ be the soft visibility map from the finest level's forward-backward check, with gradients stopped. Let $v_s$ be the area-averaged pool of $M$ to the grid of level $s$, so $v_s(p)\in[0,1]$ is the fraction of pixel $p$ that is visible. Let $r_s(p)\in[0,1]$ be a soft version of level $s$'s own native round-trip inconsistency. The photometric loss at level $s$ is then weighted by
$$w_s(p) = \max\big(\epsilon,\; v_s(p)\,(1 + \beta\, r_s(p))\big),$$
with gradients stopped through $w_s$. Here $\epsilon$ is a small floor that keeps some signal in occluded regions, and $\beta$ sets how strongly coarse mistakes on visible pixels are emphasized. The same pixels get emphasized even when their photometric error happens to be small, for example in low-contrast regions.
The design borrows from three sources. The gradient stop comes from What Matters in Unsupervised Optical Flow, which showed that networks otherwise inflate occlusion masks to cheat the loss. The non-zero floor comes from the Occlusion Avoiding loss of Improving Optical Flow and Stereo Depth Estimation by Leveraging Uncertainty-Based Learning Difficulties. That work attenuates occluded pixels rather than zeroing them, then boosts hard pixels via a focal weight on current error. It works at a single resolution and under full supervision. We replace its error-based difficulty with a disagreement-based one that exists only in a hierarchy.
Relation to Prior Work
Four papers motivate the proposal by exposing the gap. Earlier unsupervised pyramids apply round-trip masks separately at each level or at the output, with hard binary thresholds, and never ask what a coarse-level failure means:
- Occlusion Aware Unsupervised Learning of Optical Flow supervises a FlowNetS decoder at several scales with range-map occlusion masks.
- UnFlow stacks networks with a bidirectional census loss.
- UFlow, from What Matters in Unsupervised Optical Flow, dissects a PWC-Net pyramid.
Reviving Unsupervised Optical Flow brings unsupervised training to a multi-scale recurrent network, Muun-RAFT, with learned context-based upsampling between scales. It supplies our backbone and training recipe, but it keeps standard per-scale round-trip masking.
A second group supplies the multi-scale machinery and also constrains the design:
- MS-RAFT+ shows that supervising every scale from 1/16 to 1/2 beats single-scale RAFT. It reports an 18.6% lower KITTI 2015 error rate, but it is fully supervised and has no occlusion model.
- Improving Optical Flow on a Pyramid Level found that gradients from fine levels often work against coarse-level objectives, and fixed this by stopping gradients between levels.
That finding matters here. Our fine-to-coarse weights pass information from fine to coarse, so they must not become a channel for gradient interference. This is one reason the weights are gradient-stopped. We also test inter-level gradient stopping as an explicit factor.
MirrorFlow motivates treating occlusion as a property of the scene that all scales should agree on. It enforces that forward occlusion matches backward disocclusion.
Research Program
We build on the open-source Muun-RAFT release and change only how the loss is weighted. The backward pass is already computed for masking, so all levels' forward and backward flows are available. The new cost is one pooling operation and one round-trip check per level.
Early in training the finest estimate is unreliable, so the schedule has two phases. It warms up with native per-level masks and then switches to fine-level gating. We will measure the sensitivity of results to the switch point.
The program has three parts:
- Validate the signal before training with it. On the Sintel and FlyingThings3D training sets, which provide ground-truth occlusion maps, we measure how often pixels flagged "coarse-inconsistent, fine-consistent" are truly visible. We also measure how often their coarse-level error exceeds that of other visible pixels.
- Train with the weights and ablate their parts. We separate the effect of fine-level gating from the effect of the disagreement boost.
- Characterize where the idea breaks. One known weakness is that if the coarse level misses a very large motion and refinement cannot recover it, both levels fail, and the pixel is wrongly treated as occluded. We will report accuracy binned by motion magnitude so this failure mode is visible rather than hidden in averages.
Baselines
Naive reference. Muun-RAFT trained with photometric loss at every level and no occlusion handling. This measures how much any masking buys in this architecture.
Strongest existing method. Muun-RAFT with its own per-level forward-backward masks, retrained with our code on identical data, iterations (the under-100K schedule of Reviving Unsupervised Optical Flow), augmentation, and a matched tuning budget for mask thresholds. The paper reports gains of up to 28% in Sintel EPE over prior multi-frame methods, but no reported absolute Muun-RAFT figure was found on Lacuna. We will therefore report our reproduction alongside the release checkpoint.
Older unsupervised anchor. For an absolute reference, UFlow reports 11.13% error rate on KITTI 2015 in What Matters in Unsupervised Optical Flow.
Per-scale DB+OA. Jeong et al.'s Difficulty Balancing and Occlusion Avoiding weights, adapted to unsupervised photometric loss and applied independently at each level. It is the closest competitor for "attenuate occlusions, boost hard pixels." In its original supervised single-scale setting it lowers KITTI EPE from 5.04 to 4.50 on RAFT (Improving Optical Flow and Stereo Depth Estimation). Its focal strength and floor will get the same tuning budget as our $\beta$ and $\epsilon$.
Ablations. Two ablations isolate the key ingredient:
- Fine-mask only ($\beta=0$): pooled fine-level gating with no disagreement boost.
- Gradient stopping: each configuration run with and without inter-level gradient stopping.
What counts as winning. Beating the per-level-mask baseline and per-scale DB+OA requires three things:
- lower Sintel Final EPE and KITTI 2015 Fl-all, averaged over three seeds with non-overlapping spread;
- the gain concentrated in visible pixels with large motion;
- no loss on occluded pixels.
Why not just…?
Why not just use each level's own round-trip mask, as current unsupervised pyramids do? This is the default, and it is exactly what our diagnostic targets. If coarse-level inconsistent pixels turn out to be overwhelmingly true occlusions on Sintel's ground truth, the default is fine and our proposal fails at the first step. The per-level-mask baseline settles it in training as well.
Why not just downsample the final mask and stop there? This is our $\beta=0$ ablation, and it is a serious threat. It already fixes coarse levels discarding visible-but-misestimated pixels, which may be most of the benefit. The disagreement boost has to earn its place with a measurable further gain on large-motion visible pixels. If it does not, the simpler rule is the recommendation, and the paper reports that.
Why not just apply Jeong et al.'s focal difficulty weighting at every scale? The difficulty cue of Improving Optical Flow and Stereo Depth Estimation is current error magnitude. Under supervision that is the true error; without labels it becomes photometric residual, which is inflated by lighting changes and deflated in textureless regions where wrong flow still matches. Cross-scale disagreement avoids both problems. This is an argument, not evidence, so the matched-budget per-scale DB+OA baseline and a textureless-region breakdown decide it.
Why not just replace round-trip checks with synthetic occlusions? Imposing Consistency for Optical Flow Estimation trains occlusion handling from pasted random masks. It reports 4.33% Fl-all on KITTI 2015, but in a semi-supervised setting that is not comparable to ours. Pasted blobs do not reproduce parallax or non-rigid occlusion, and the same group returned to cycle consistency in later work. Synthetic masks are complementary rather than a substitute, so we do not claim superiority over them.
Proposed Experiments
Training follows the Muun-RAFT unsupervised protocol on Sintel and KITTI raw sequences. Evaluation covers four benchmarks:
- Sintel Clean and Final, by EPE, split into occluded and visible pixels and binned by motion magnitude;
- KITTI 2015, by Fl-all;
- KITTI 2012, by average endpoint error;
- Spring, for cross-dataset generalization.
The first decisive check runs before any training. On Sintel's ground-truth occlusion maps, if the "coarse-inconsistent, fine-consistent" set is not mostly visible pixels with above-average coarse error, the disagreement boost is abandoned.
The second decisive check is the $\beta=0$ versus $\beta>0$ comparison, restricted to visible pixels moving more than 40 pixels. This is where the claim predicts its gain and where any gain must show up.
The third check crosses each configuration with and without inter-level gradient stopping. It tests the unresolved question raised by Improving Optical Flow on a Pyramid Level: whether isolating levels helps or hurts once masks flow from fine to coarse.
All runs report training time and memory. The claim that the method costs one extra pooling operation should be verified, not assumed.
References
- What Matters in Unsupervised Optical Flow
- Improving Optical Flow and Stereo Depth Estimation by Leveraging Uncertainty-Based Learning Difficulties
- Occlusion Aware Unsupervised Learning of Optical Flow
- UnFlow: Unsupervised Learning of Optical Flow With a Bidirectional Census Loss
- Reviving Unsupervised Optical Flow: Concept Reevaluation, Multi-Scale Advances and Full Open-Source Release
- MS-RAFT+: High Resolution Multi-Scale RAFT
- Improving Optical Flow on a Pyramid Level
- MirrorFlow: Exploiting Symmetries in Joint Optical Flow and Occlusion Estimation
- Imposing Consistency for Optical Flow Estimation