Video frame interpolation means generating the frames that would have been captured between two real ones, for example to turn 30 frames per second into slow motion. The newest methods are "continuous-time": they can be asked for a frame at any moment $t$ between the two inputs, such as $t=0.37$, rather than only at the midpoint. They work by estimating motion between the two frames and then moving pixels to where they should be at time $t$. On 4K video with very fast motion these methods still fail. The usual explanation is that the motion estimator only looks at a small neighborhood of each pixel, so it misses objects that moved hundreds of pixels. We argue there is a second, separate failure that becomes worse at arbitrary times. When motion is pushed forward to time $t$, some regions are left with no motion estimate at all: these are holes where background is newly uncovered. Their width grows with both the size of the displacement and $t$. Filling them requires motion information from far away, and the small convolutional network that cleans up the motion field cannot see that far.
This note proposes to separate the two failures, measure each, and fix the second one with a cheap long-range motion-completion step that operates at the queried time $t$. Its reach is set by the displacement and by $t$, and its cost is linear in image size. If the decomposition holds, it explains why upgrading the motion estimator alone cannot close the large-motion gap. It also gives a targeted, low-cost fix for deterministic interpolators, the kind that avoid the latency and texture hallucination of diffusion-based models.
Core Claim
The closest existing system is GIMM. It encodes initial two-way motion estimates into a motion feature map, forward-splats that map to time $t$ (pushes each feature to its moved position), refines it with a local network, and decodes motion at any $(x, y, t)$ with a small coordinate MLP. GIMM's authors note that it cannot recover when the initial motion estimator fails on large displacements. The research file's Gap 1 recommends dropping non-local attention into GIMM's refiner. We think that framing merges two different problems.
The first problem is anchor-motion error: the motion estimated between the two real frames is wrong. A better estimator with global matching addresses this directly. NODE-Flow, which uses Transformer blocks inside a continuous-time motion refinement, reports KITTI 2015 endpoint error of 2.38 against 2.85 for its global-matching baseline. The second problem is time-$t$ completion error. Even with perfect anchor motion, splatting to time $t$ leaves holes, and a hole of width $w(t)$ grows roughly as $t\,|d|$ for displacement $d$, because the uncovered region widens linearly as the occluder moves. A local refiner with receptive radius $r$ cannot see across a hole wider than about $2r$. It then has to invent motion inside the hole, which produces the tearing and ghosting that TOFlow traced to uncovered regions that need context from distant pixels.
Our claim is that, on large-motion 4K benchmarks, a substantial share of the remaining error of continuous-time interpolators sits in these time-$t$ holes. That share does not go away when the anchor motion estimator is upgraded. A completion step whose reach scales with $t\,|d|$ removes it at little cost.
Research Program
The first part is a diagnostic study. We use X4K1000FPS, whose 1,000 FPS capture provides dense real intermediate frames at many values of $t$ (see XVFI). For each test pixel we compute two quantities: whether it lies in a splatting hole at time $t$, and its distance $\delta$ to the nearest pixel with a valid motion estimate. We then plot the error of GIMM against $\delta$ under three anchor-motion conditions. The first is GIMM's own estimator. The second is a stronger global-matching estimator. The third is a pseudo-oracle: anchor motion computed by chaining motion across the dense 1,000 FPS frames, where each step is small and therefore reliable.
The decomposition predicts a specific pattern. Error should rise sharply once $\delta$ exceeds the refiner's measured receptive radius, and it should stay high even with the pseudo-oracle. If error instead collapses under the pseudo-oracle, the second failure is negligible, and the honest conclusion is that a better motion estimator is all that is needed. This study is the first decision point.
The second part is the completion module. After splatting to time $t$, each hole location gathers motion features along the line through it in the direction of the surrounding anchor motion. That is the direction along which the occluder moved, so it is where the relevant uncovered background and occluder motion lie. Samples are spread over a span proportional to $t\,|d|$, with a fixed number $K$ of samples, and are fused by attention. Two axis-aligned row and column passes, in the style of CCNet's criss-cross attention, provide a global fallback where the motion direction is unreliable.
This differs from the deformable sampling in DAS and from TMNet's $t$-modulated deformable convolution. In those methods the offsets are learned and stay near the kernel. In ours, the sampling span is set explicitly by $t$ and the displacement and can reach hundreds of pixels at full resolution. The design also responds to the file's point that criss-cross routing assumes straight horizontal and vertical paths, which do not follow motion trajectories: we route primarily along the motion direction.
The module is inserted only into GIMM's motion-latent stage. It is trained end-to-end at random $t$, unlike TMNet's two-stage schedule. Cost is $O(NK)$ for $N$ pixels.
Baselines
The naive reference point is linear scaling of two-way motion, which moves pixels a fraction $t$ of the way along the estimated motion. GIMM reports 30.09 dB PSNR for this approach on Vimeo-Septuplet-Flow 6× interpolation, against 30.45 dB for GIMM itself. We also include recursive midpoint interpolation, which generates the midpoint and then repeats on each half. We run it with VFIT, a midpoint-only Transformer interpolator reporting 36.96 dB on Vimeo-90K, applied repeatedly at dyadic times. This shows how much continuous-time querying is worth on our splits.
The strongest existing deterministic method is GIMM-VFI-F, which reports 32.91 dB PSNR on XTest-2K against 32.53 dB for EMA-VFI. We retrain it from the released recipe on the same data and with the same step count. This guards against reproduction gaps, and both the reported and reproduced numbers will be shown. We also include XVFI, which handles large motion with a multi-scale pyramid. It is the pyramid alternative, run with its published checkpoint and retrained on our split. The file reports only its +2.6 dB margin over DAIN on X-TEST, not an absolute value.
The strongest competitor on our specific question is GIMM with its anchor estimator replaced by a global-matching estimator, matched to our variant in parameters and compute. No reported VFI result for this configuration was found on Lacuna, so we will establish it ourselves.
Two ablations isolate the key ingredient. The first enlarges GIMM's local refiner, with dilations and extra depth, to the same FLOPs as our module. This tests whether added capacity alone explains any gain. The second keeps our module but samples a fixed span that does not depend on $t$, which tests whether scaling the reach with $t\,|d|$ matters.
Beating the strongest baseline means a full-frame PSNR gain on XTest-2K over global-anchor GIMM, plus a clearly larger gain on hole pixels with $\delta$ greater than the refiner radius. All runs use identical training data (Vimeo-Septuplet plus X4K training clips), identical numbers of hyperparameter trials, and three seeds.
Why not just…?
Why not just swap in a global-matching motion estimator? This is the real threat, and the diagnostic study tests it directly. A global estimator fixes anchor motion. It never sees time-$t$ splatting holes, because those holes exist only after the motion is pushed forward to the queried time. If pseudo-oracle anchor motion removes most of the large-$\delta$ error, then this alternative is sufficient and we will report that result rather than the module. Our bet is that it will not, and the stratified plot will show which is true.
Why not just use a multi-scale pyramid, as XVFI does? Pyramids extend reach by downsampling. The research file records that XVFI's coarse passes still struggle with fine textures and delicate boundaries, and Li et al. document how downsampling discards detail and dilutes long-range context. We include XVFI and also measure error near occlusion boundaries specifically. Hole boundaries are where a coarse motion estimate would be most visibly wrong.
Why not just use full attention or a large diffusion Transformer? Full pairwise attention scales quadratically. CCNet reports 11× memory and 85% FLOP savings from axis-factorized attention, and ArbInterp has to split long sequences into segments even with a 1.3B-parameter model. ArbInterp reports FVD 319.9 at 32× on MultiInterpBench, a distribution-level score that does not reward pixel-exact reconstruction. We will report FVD alongside PSNR on MultiInterpBench and also report latency, so the trade-off between the two kinds of methods is visible.
Proposed Experiments
The main evaluation is XTest-2K and X-TEST, where displacements reach 196.5 pixels. We report PSNR and SSIM over the full frame, over hole pixels, and in bins of $\delta$ and $t$, including non-dyadic times. Secondary evaluations cover Vimeo-Septuplet-Flow 6×, to confirm there is no regression on moderate motion, and SNU-FILM-arb, for which no reported number was found on Lacuna. MultiInterpBench at 8×, 16× and 32× tests multi-scale arbitrary interpolation. If access is granted, the 2,000 FPS chromatic-spikes data serves as an out-of-distribution test with dense real ground truth.
Two checks are decisive. The first is the pseudo-oracle decomposition: what share of large-$\delta$ error survives perfect anchor motion. The second is whether our module's gain over global-anchor GIMM grows with $t\,|d|$ while the matched-FLOP local refiner and the fixed-span variant do not show that growth. A flat gain curve would mean the reach-scaling idea is wrong, even if average PSNR improves.
References
- Generalizable Implicit Motion Modeling for Video Frame Interpolation
- Learning Optical Flow Field via Neural Ordinary Differential Equation
- Video Enhancement with Task-Oriented Flow.
- XVFI: eXtreme Video Frame Interpolation
- CCNet: Criss-Cross Attention for Semantic Segmentation
- DAS: A Deformable Attention to Capture Salient Information in CNNs
- Temporal Modulation Network for Controllable Space-Time Video Super-Resolution
- Video Frame Interpolation Transformer
- Non-locally Enhanced Encoder-Decoder Network for Single Image De-raining
- Arbitrary Generative Video Interpolation