Recurrent video restorers such as BasicVSR++ carry information from frame to frame by predicting where each pixel moved. They then pull the matching features forward. The operators that predict this motion are local. The optical flow network searches a small neighborhood at each level of a coarse-to-fine image pyramid. The deformable convolution that refines the flow is a 3×3 filter whose sampling points can shift, but only by a learned residual around the flow estimate. When an object jumps farther than these operators can see, the alignment is wrong. The error then travels down the recurrent chain, and the output shows it as flicker, crawling texture, or ghosting. We propose a cheap global matching step that is invoked only where a consistency check says local alignment has failed. The step searches entire rows and columns of the neighboring frame at full feature resolution, alternating between the two axes twice, so any pixel can reach any location, as in two-pass criss-cross attention. It returns a single hard correspondence rather than a blurry average.
The claim is that much of the temporal instability of strong recurrent restorers on fast motion is caused by alignment errors that a local receptive field cannot recover from. A global but factorized search, run at full resolution and gated by failure detection, should remove these errors without the blur of soft non-local averaging or the detail loss of downscaled attention. If the claim holds, restorers become temporally stable on large-motion footage at a small added cost. If it fails, the experiments will say whether the fault lies with receptive field size or somewhere else, such as degradation of the matching features.
Core Claim
Three prior lines of work each solve part of this problem. ERDN shows that shrinking receptive fields at the high-resolution levels of an alignment pyramid break deformable offset estimation under large blur. It fixes this with dilated convolutions that keep an equivalent receptive field across levels, and reports 33.31 dB on DVD. Its reach is still a fixed sparse grid, however, and it works in a sliding window rather than a long recurrent chain. MANA shows that hard one-hot cross-frame matching beats soft attention on erratic motion, gaining 1.87 dB over EDVR on Parkour. Its search is confined to a 9×9 window, and it trails alignment-based recurrent models on moderate-motion benchmarks. PFNL makes spatio-temporal non-local blocks affordable by downscaling features inside the block. That discards the sub-pixel cues that recurrent super-resolution depends on.
Our proposal takes something from each. From MANA we take hard matching. From CCNet we take the idea that two axis-factorized passes give full global reach at roughly $O(N\sqrt{N})$ cost, where $N$ is the number of pixels. CCNet was developed for single-image segmentation, not for cross-frame correspondence. From flow-guided deformable alignment in BasicVSR++ we take the idea that the global match should only supply a base offset, with a local deformable residual refining it. The new element is the combination, placed inside a bidirectional recurrence and triggered selectively. We are not proposing a new attention block to add everywhere. We are proposing a way to catch and repair the alignment failures that feed the "snowball effect" documented by Revisiting Temporal Alignment for Video Restoration.
Research Program
The base system is BasicVSR++ with second-order bidirectional propagation and flow-guided deformable alignment. At each propagation step, we compute a per-pixel failure score from two signals. The first is the disagreement between forward and backward flow. The second is the patch-feature dissimilarity between the current frame and the warped neighbor, in the spirit of the non-parametric re-weighting used by Revisiting Temporal Alignment.
Where the score is high, the global search runs as follows. For a pixel $p=(x,y)$ in frame $t$, it scores every position along row $y$ of frame $t-1$ and keeps the best horizontal position $x_1$. It then scores every position along column $x_1$ and keeps the best vertical position $y_1$. It repeats the row-then-column search once more starting from $(x_1, y_1)$. Two alternations suffice to reach any location, for the same reason that two criss-cross passes give CCNet a full receptive field.
Matching is done on features taken before the deformable refinement. The search runs at the resolution at which propagation operates, with no downscaling. Selection uses a soft-argmax with an annealed temperature during training and a hard argmax at inference.
The resulting displacement replaces the flow estimate as the base offset for the deformable convolution. The replacement is soft: a learned gate driven by the failure score blends the flow and global-match offsets. In the unaffected regions, the model reduces exactly to BasicVSR++.
Training uses the standard reconstruction loss. We also add the short-term warping loss of Learning Blind Video Temporal Consistency, computed only on non-occluded pixels. The analysis side of the program measures three things. It checks where BasicVSR++'s temporal errors concentrate, as a function of true displacement magnitude. It checks whether those errors persist or grow over subsequent recurrent steps. It checks whether the gated global search removes them.
Baselines
The naive reference point is frame-by-frame restoration with a strong single-image model, Restormer. It is retrained on the same data and scale factor, and it shows the flicker floor when no temporal information is used.
The strongest general competitor is BasicVSR++. It is the exact architecture we modify, and its reported advantage is 0.82 dB PSNR over BasicVSR and 1.3 dB over EDVR on REDS4, with 65% fewer parameters (no absolute REDS4 value is recorded in our file). The strongest method specifically targeting alignment error accumulation is the iterative alignment approach of Revisiting Temporal Alignment. It reports 37.84 dB PSNR on Vimeo-90K, and raised IconVSR from 37.47 to 37.56 dB as a plug-in. Its REDS4-Fast numbers are described only qualitatively; there is no reported result on Lacuna for its REDS4-Fast PSNR. For extreme motion, the reference is MANA, which leads EDVR by 1.87 dB on Parkour.
All of these are retrained by us on the same training split with identical schedules, crop sizes, and tuning budgets. Where possible, our variant is matched in parameters and FLOPs by removing residual blocks. Beating BasicVSR++ means two things on REDS4-Fast and Parkour: a lower temporal warping error and a PSNR at least equal to it. It also means being within 0.05 dB on Vimeo-90K and Vid4, where motion is moderate and the gate should rarely fire.
Three ablations isolate the key ingredient. The first is a restricted-reach variant that runs the identical hard-matching module in a 9×9 window, as in MANA. The second is a downscaled variant that runs full-image dense matching at 1/4 resolution, as in PFNL, at matched FLOPs. The third is an always-on variant with no gate.
Why not just…?
Why not just enlarge the local search window? A 31×31 or 63×63 window with hard matching might catch most large motions. If displacements in the test sets are bounded, it could match our method. The restricted-reach ablation sweeps the window size. We will plot accuracy against the true displacement distribution, taken from flow computed on the clean ground truth. The global search is justified only if errors remain beyond the largest affordable window. If a 31×31 window recovers nearly all of the gain, we will report that and drop the global claim.
Why not just do global matching at coarse resolution and refine locally? This is the most serious threat. SPyNet already works coarse-to-fine, and large motions are coarse by nature. Our counter-hypothesis is that the objects causing failures in fast footage, such as thin limbs in Parkour or blurred edges in DVD, disappear or merge at coarse scales. That is the same loss of fine structure through downsampling that Guanbin Li et al. document. The FLOP-matched downscaled ablation settles this directly. We will also break results down by object size. If coarse matching ties us overall and on small structures, the full-resolution argument fails.
Why not just post-process with a blind consistency filter? The ConvLSTM filter of Lai et al. runs at 418 FPS, and Deep Video Prior needs no training data. Both only smooth an output that has already been restored, though. Lai et al. note that the filter blurs jittery inputs. DVP costs about 2 seconds per frame and has an unstable stopping point. Neither can recover detail that misalignment destroyed. We will apply Lai et al.'s filter to BasicVSR++ outputs and compare both warping error and PSNR. Winning on warping error alone would not count.
Proposed Experiments
We train for 4× super-resolution on REDS and Vimeo-90K. We evaluate on REDS4, REDS4-Fast, Vimeo-90K, Vid4, and Parkour. We also run a deblurring transfer on DVD against ERDN's reported 33.31 dB. Metrics are PSNR, SSIM, LPIPS, and a temporal warping error of consecutive outputs under flow computed on the ground truth, as in Lai et al. Runtime is reported in milliseconds per frame, against BasicVSR's 63 ms and MANA's 251 ms as reported.
The first decisive check is a stratified analysis. We bin pixels by true displacement magnitude and measure BasicVSR++'s warping error and the lifetime of its errors across recurrent steps in each bin. The proposal is motivated only if errors concentrate beyond roughly the local operators' reach and persist over later frames.
The second check is the ablation triangle of restricted reach, downscaled global, and full-resolution global at matched cost. Our claim requires the full-resolution global variant to win specifically in the large-displacement, small-structure bins.
The third check is gate firing rate. The gate should fire on a small fraction of pixels on Vid4 and Vimeo-90K and substantially more on REDS4-Fast and Parkour. That would show the module repairs failures rather than acting as generic extra capacity.
The fourth check injects a synthetic misalignment at frame $k$ and tracks the error over frames $k+1$ through $k+20$. With the global rescue, the error should decay, while in BasicVSR++ it should persist. This isolates the temporal-consistency mechanism from overall reconstruction quality.
References
- ERDN: Equivalent Receptive Field Deformable Network for Video Deblurring
- Memory-Augmented Non-Local Attention for Video Super-Resolution
- Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations
- CCNet: Criss-Cross Attention for Semantic Segmentation
- BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment
- Revisiting Temporal Alignment for Video Restoration
- Learning Blind Video Temporal Consistency
- Restormer: Efficient Transformer for High-Resolution Image Restoration
- Non-locally Enhanced Encoder-Decoder Network for Single Image De-raining
- Blind Video Temporal Consistency via Deep Video Prior