Plane-Routed Attention for Filling Disocclusions in Dynamic Video

When a camera shifts sideways, or a car drives out of the frame, surfaces that were hidden come into view. A system synthesizing that new view has no pixels for them, and these revealed regions are called disocclusions. Video models usually fill them by attending across many frames, looking for a frame where the hidden surface was visible. Standard attention compares every image patch with every other patch, so its cost grows with the square of the number of patches. That cost is why this approach is slow. A large part of the revealed content in driving and street video, however, sits on one flat surface such as a road, a floor, or a building front. For a flat surface, the mapping between any two frames is a single 3×3 matrix called a homography, so we already know where to look. This proposal uses that fact to decide where attention is spent. Patches on the dominant plane attend only to a small window around the location the homography predicts in each other frame. The more expensive global attention is reserved for patches that leave the plane, either because they stick out of it in 3D (parallax) or because they move independently.

The closest existing system is M2SVid. It converts monocular video to stereo, and its "Selective Full Attention" sends patches inside a disocclusion mask to full cross-frame attention while every other patch uses cheaper factorized attention. We keep that idea and add geometry. M2SVid's routing asks whether a patch is a hole. Our routing asks whether geometry already tells us where the answer is. The claim is that, in scenes with a dominant plane, geometry-routed attention fills disocclusions at least as well as mask-routed full attention. It should also cost a fraction of the compute, with that fraction roughly proportional to how much of the hole lies off the plane. Finally, it should flicker less, because the correspondences for planar regions come from geometry rather than from learned similarity.

Core Claim

Planar-parallax geometry splits the motion of every pixel between two frames into two parts: the motion explained by a homography of a reference plane, and a residual along epipolar lines (the lines on which a point's match must lie) caused by height above or below that plane. DepthP+P and MonoPP use this split to cancel camera rotation and recover metric depth on roads. Both treat moving objects as corruption to be masked out, and neither fills holes. Deep Homography Estimation for Dynamic Scenes estimates the background homography robustly by predicting a mask of moving pixels. It then discards those pixels and does not reconstruct what they hide.

We reuse these methods to supply geometry, and we put their output to a different use: as a routing signal for attention inside a video inpainting and refinement model. A patch whose residual is small and that is not flagged as moving is "planar-static." Its content in frame $s$ is predicted at $H_{t\to s}p$, where $H_{t\to s}$ is the frame-pair homography and $p$ is the patch location. It attends only within a $w\times w$ window there. The remaining fraction $\rho$ of patches receive hierarchical sparse global attention. Per layer, the cost becomes roughly

$$C \approx N T w^2 + \rho\, N T \log(NT),$$

where $N$ is the number of patches per frame, $T$ the number of frames, and the logarithmic term comes from log-linear sparse attention (LLSA). Full attention would cost $(NT)^2$. The prediction to test is that quality stays flat while compute tracks $\rho$.

Research Program

The system has three parts.

The geometry front end estimates a dominant-plane homography per frame pair. On driving data it follows DepthP+P's ground-plane alignment with known camera height. Elsewhere it uses an MHNm-style network with a dynamics mask, from Deep Homography Estimation for Dynamic Scenes. The front end then computes a per-pixel parallax residual from a monocular depth estimate and flags independently moving pixels where the residual violates epipolar consistency. This is the same signal DepthP+P treats as corruption; here it is turned into a feature.

The second part is a video refinement backbone. We start from the single-step video-diffusion refiner architecture of M2SVid and replace its attention layers with the routed variant. Planar-static patches get homography-warped windows. Off-plane and moving patches get LLSA-style hierarchical top-k attention over all frames, so that the routing itself never needs a full $N\times N$ similarity matrix. LLSA identifies that selection step as the hidden quadratic cost in ordinary top-k attention.

The third part is a set of analysis tools. We stratify every evaluation by planar fraction and by the share of the disoccluded area that lies on the plane, so that the claimed scaling relation can be checked directly rather than inferred from averages.

The cited work plays three distinct roles. M2SVid motivates the project, because it shows that targeted attention fixes disocclusion holes. DepthP+P, MonoPP and Le et al. supply the geometry. LLSA supplies the sparse mechanism. Two papers act as constraints. SAFF shows that, without structural priors, monocular dynamic decomposition yields "fluffy" boundaries; this argues for explicit geometry over learned similarity at boundaries. Jiang et al. document ghosting when a single homography is forced onto multi-depth content. That is exactly why off-plane patches must escape the window rather than be warped.

Baselines

The naive reference is geometric warping alone. The source frame is forward-warped with estimated depth, the remaining holes are filled by homography-warped copies from neighbouring frames, and anything left over is filled by a standard image inpainter (Generative Image Inpainting With Contextual Attention). This reference tells us how much of the problem is solved by geometry with no learning at all.

The strongest existing method is M2SVid with its mask-routed Selective Full Attention. It is the only system in the file that attacks exactly this cost–quality trade-off for disocclusion in video. Its paper reports 16 frames at 512×512 in 2.1 s on an A100, which it describes as 6× faster than StereoCrafter. It also reports an average user-study rank of 1.43 among 4 methods. The file states that M2SVid has the highest PSNR and LPIPS on Stereo4D, but no reported result is found on Lacuna for the actual Stereo4D PSNR or LPIPS values. We will therefore retrain M2SVid and our model from the same initialization, on the same data, with the same number of steps and the same hyperparameter search budget, and report both. Beating it means two things. First, matching or exceeding its disocclusion-region PSNR and LPIPS. Second, cutting attention FLOPs and wall-clock time by at least 2× on plane-dominated clips, or improving temporal consistency at equal compute.

A third baseline is flow-guided propagation, as in Deep Flow-Guided Video Inpainting, followed by the same refiner using M2SVid's attention. This separates the question "geometry inside attention" from the simpler "geometry before attention."

Two ablations isolate the key ingredient, and both keep the routed budget ($\rho$ and $w$) fixed. In the first, homography-warped windows are replaced by unwarped windows at the same image coordinates, which tests whether the warp itself matters. In the second, the parallax-based routing is replaced by a learned saliency score in the style of SG-Former, which tests whether geometry routes better than data-driven importance.

Why not just…?

Why not just warp neighbouring frames with the homography and copy the pixels, with no attention at all? For planar holes this is a serious competitor, and we expect it to come close. Our method includes it as the "free" path. The open question is whether the learned refinement still adds value at plane boundaries, in shadowed regions, and on revealed off-plane structure such as curbs, poles and parked cars. On off-plane structure a single homography produces ghosting (Jiang et al.). Split4D notes that removing a moving object also leaves illumination and shadow residue, which copying cannot fix. The naive baseline above, scored separately on planar and off-plane hole pixels, settles this. If copying matches us on the off-plane pixels too, then the learned refiner is unnecessary and the paper becomes a negative result.

Why not just use optical flow or dense depth for correspondence everywhere, since that covers the planar case too? Flow is least reliable precisely at occlusion boundaries and under independently moving objects, which is where disocclusions sit. A homography, by contrast, is fit from thousands of background pixels and remains stable when a dynamics mask excludes movers (Deep Homography Estimation for Dynamic Scenes). This is an empirical claim rather than a theorem, and the flow-guided baseline tests it directly. If flow-guided windows match homography windows, we will report that the planar prior adds nothing beyond robustness on difficult clips.

Why not just use M2SVid's mask routing combined with a generic efficient attention such as LLSA or Swin-style local windows? Generic sparse attention reduces cost without knowing where the answer is. Local windows lose the cross-frame reach that disocclusion filling needs, which is SG-Former's argument against Swin-style windows. Top-k selection must still score candidates, a step LLSA shows is quadratic unless it is made hierarchical. Our off-plane path already uses LLSA, so this alternative is the "learned-saliency routing" and "unwarped window" ablations at equal budget. If those match the full model, the planar contribution is null.

Proposed Experiments

The main testbed is KITTI. Its stereo pairs give a real ground-truth right view for monocular-to-stereo synthesis, the ground plane dominates, and the clips contain moving cars. DepthP+P reports 0.134 Abs Rel monocular depth there, which we use as a sanity target for the geometry front end. Stereo4D and Ego4D, following M2SVid's protocol, provide less planar footage. Homography accuracy under dynamics is checked on VidSet_d, where MHNm reports corner error below 1 pixel on more than 85% of clips.

Quality is measured by PSNR and LPIPS restricted to disoccluded pixels, split into on-plane and off-plane pixels. Temporal flicker is measured as warping error between consecutive synthesized frames. Cost is measured as attention FLOPs and A100 latency at 16 frames and 512×512, matching M2SVid's reported setting.

Two checks are decisive. The first is a plot of compute and quality against measured off-plane fraction $\rho$ across clips. The claim holds only if cost rises roughly linearly with $\rho$ while the quality gap to M2SVid stays near zero. The second is an indoor and non-planar stress test, using Stereo4D clips binned by planar fraction. There, the router should fall back to sparse global attention on most patches and recover M2SVid-level quality. If quality instead drops below M2SVid, the routing is causing harm rather than gracefully degrading, and the method should be restricted to plane-dominated domains.