Many video deblurring systems run frame by frame. At each step they estimate how pixels moved since the previous frame, pull the previous sharp result into alignment, and feed it forward through a recurrent memory. Each pull only searches a small window around the estimated motion. The best example is a 5×5 "pixel volume", meaning a set of 25 candidate source pixels from which the network blends. When the true source pixel lies outside that window, because the object moved fast or the motion estimate was wrong on a blurry frame, the network must blend wrong pixels. That error enters the recurrent memory and is carried into later frames.
We propose to test one specific claim. Much of the error that propagates through recurrent deblurring starts at these "escape" events, where the true match falls outside the local search window. We further claim that a small, compute-matched budget of globally retrieved candidates, plus an explicit "no good match" option, removes most of that propagated error. A larger network or full-frame attention should not be needed for this. The deliverables are a measurement protocol for how long an alignment error persists in the recurrent state, and a minimal alignment module that changes how a fixed candidate budget is spent.
Core Claim
Three lines of prior work each touch one part of this failure.
The first line couples motion and restoration. Recurrent Video Deblurring with Blur-Invariant Motion Estimation and Pixel Volumes (PVDNet) estimates motion with a blur-invariant network. It then aligns softly over a 5×5 candidate volume and reports 32.31 dB PSNR on the Su et al. dataset with 10.5M parameters. The same paper notes a "first-frame problem": early poor outputs contaminate the recurrent state.
The second line enlarges what the network can see. ERDN shows that keeping receptive fields consistent across pyramid scales gives a 0.99 dB gain on the 1000 fps HFR-DVD benchmark, which has large displacements. This suggests that search range, not model capacity, limits performance under fast motion.
The third line documents compounding errors. Cascaded feedback in Cascaded Deep Video Deblurring Using Temporal Sharpness Prior (CDVD-TSP) re-estimates flow from intermediate deblurred frames. Its authors acknowledge that this breaks down when every neighboring frame is uniformly blurred. Mitigating Sequential Error Propagation in Multi Stage Pipelines argues more generally that downstream stages treat upstream artifacts as real detail.
None of these papers measures propagation directly. None separates errors caused by out-of-window matches from errors caused by other sources. None asks whether a fixed candidate budget is better spent locally or globally. That question is what our proposal is built around.
The mechanism replaces PVDNet's 25 local candidates with the same number of candidates, allocated differently. Nine are local: a 3×3 window around the estimated flow. Sixteen are global: the top matches from a coarse all-pairs correlation computed at 1/8 resolution. One extra slot is a null option that falls back to the current frame's own features. For a pixel $x$ at frame $t$, the aligned feature is
$$a_t(x) = \sum_{c \in L(x) \cup G(x)} w_c(x)\, h_{t-1}(c) \;+\; w_\varnothing(x)\, f_t(x).$$
The symbols are defined as follows. $h_{t-1}$ is the previous recurrent state. $L(x)$ is the 3×3 local set around the flow estimate. $G(x)$ is the 16 retrieved global locations. $f_t$ is the current frame's encoded features. The weights $w$ are a softmax over all 26 options. The null weight $w_\varnothing$ also scales the recurrent update, so that a frame with no trustworthy match writes less into the memory.
Motion and restoration stay coupled through the recurrence. Motion is estimated between the previous restored frame and the current blurry input. Following Motion-Compensated MR CINE Reconstruction With Reconstruction-Driven Motion Estimation, which drives a motion network only by the final reconstruction loss, we train without any photometric warping loss. That paper's argument is that a warping loss forces the motion estimator to match corruption patterns. We also test that argument in this setting rather than assume it transfers from MRI.
Research Program
The first component is a propagation measurement. It is the part we expect to be useful even if the module fails.
GOPRO is built from 240 fps sharp frames. On those sharp frames we can compute reference flow with any off-the-shelf estimator. A pixel counts as an "escape" pixel when its reference displacement differs from the model's own flow estimate by more than the window radius: 2 px for 5×5, 1 px for 3×3. For each model we then record two things. One is the per-frame error at escape pixels. The other is an error persistence curve: the excess error at the same scene points, tracked forward along reference flow for the next $k$ frames.
We will also inject controlled escape events. We do this by dropping frames from a sequence, which doubles or triples the displacement at a known time step. We then measure the number of frames until the excess error falls below half its initial value. This gives a direct, comparable quantity for "sequential error propagation", which prior papers only describe qualitatively.
The second component is the module itself, built as a drop-in change to PVDNet's alignment step. We keep PVDNet's backbone fixed so that any gain is attributable to the allocation of candidates.
The third component is a stress test on the failure mode no window size can fix. This is uniform blur with no sharp anchor anywhere in the neighborhood, the case CDVD-TSP's authors acknowledge. We expect the null option to keep the memory from being corrupted in these sequences, but not to restore detail. We will report that honestly as a boundary of the method.
Baselines
The naive reference point is PVDNet retrained in our pipeline with its original 5×5 local volume. It shares our backbone, data and schedule, so it isolates the alignment change. Below it sits a no-alignment variant that stacks frames without motion. Deep Video Deblurring and VDFlow both show that this variant trails flow-based alignment; VDFlow reports a 0.70 dB average gap. We include it to anchor how much temporal aggregation contributes at all.
The strongest existing method on the Su et al./DVD data is ERDN, which reports 33.31 dB PSNR. It is the right competitor because it attacks the same large-displacement problem by widening receptive fields rather than reallocating candidates. One caution: ERDN reports EDVR at 28.51 dB on DVD, while PVDNet reports EDVR at 31.82 dB on the Su et al. data. Cross-paper numbers on this benchmark therefore come from different protocols. We will retrain ERDN, PVDNet and CDVD-TSP from released code under one protocol and report both the published and the retrained figures.
The relevant published reference points are as follows. CDVD-TSP reports 31.67 dB on GOPRO. MemDeblur, in Multi-Scale Memory-Based Video Deblurring, reports 31.77 dB on downsampled GOPRO. Pay Attention to Hidden States for Video Deblurring (PAHS) reports 34.27 dB on REDS. For ERDN on HFR-DVD, only a relative 0.99 dB gain is available, not an absolute number. No reported result was found on Lacuna for PAHS or ERDN absolute PSNR on GOPRO.
All methods will be matched on training data, iterations and hyperparameter search budget. We will report multiply-accumulate operations per frame and runtime alongside PSNR.
Beating the baselines means three things. First, our module must beat retrained PVDNet at equal multiply-accumulate count, with the gain concentrated on escape pixels and HFR-DVD. Second, it must shorten the error persistence half-life. Third, it must match or exceed retrained ERDN on DVD and HFR-DVD at lower compute. If the global candidates help only on full-frame PSNR and not on escape pixels, the core claim is falsified.
The key ablations remove one ingredient at a time. We test 25 local candidates against 9 local plus 16 global. We remove the null option. We add a photometric warping loss back in. Each ablation is run at a fixed candidate count.
Why not just…?
Why not just enlarge PVDNet's window to 9×9 or 15×15? This is the most serious threat. If escape pixels mostly sit just outside 2 px, a modestly larger window will catch them. Cost grows with the square of the radius: 81 or 225 candidates against our 26. We will run 7×7 and 9×9 windows. We will also histogram escape distances on GOPRO and HFR-DVD. If most escapes fall within 4 px and a 9×9 window matches our method at its own cost, we will report that a wider local window is the right fix and that global retrieval is unnecessary.
Why not just use non-local attention or a memory bank, as PAHS and MemDeblur do? Both already retrieve matches beyond the convolution window. PAHS attends across the full frame with a learned gate. MemDeblur retrieves sharp features from up to 50 frames back. Both, however, replace explicit motion with attention matching, at a cost that grows with frame size. Neither reports how long alignment errors persist. Our hypothesis is that a flow-anchored local set plus a handful of global candidates gets most of the benefit at a fraction of the cost. The escape-pixel and persistence measurements, run on PAHS and MemDeblur as well, will show whether their attention actually shortens persistence more than our method does.
Why not just get better flow, or process bidirectionally? If flow were accurate, a 5×5 window would suffice. The escape labels separate two cases. In the first, flow was wrong, which better motion estimation would fix. In the second, flow was right but the 5×5 window was still too small for the displacement, for example on thin or fast objects. We will swap in the strongest flow estimator we can run within budget, with PVDNet's original window. If that alone closes the gap, our candidate reallocation is not the right lever. A bidirectional pass addresses the first-frame problem but not escape events in mid-sequence. We will test it as an orthogonal addition rather than an alternative.
Proposed Experiments
We will train on GOPRO and on the Su et al. DVD data, and evaluate on GOPRO, DVD, HFR-DVD and REDS. We will also run BSD to check that conclusions hold on real captured blur rather than only on blur synthesized by averaging frames.
The decisive checks are three. First, at matched multiply-accumulate count, does the local-plus-global allocation reduce error specifically on escape pixels, while leaving non-escape pixels unchanged? Second, after a controlled frame-drop event, does excess error decay faster? A shorter half-life is the direct evidence that the method stops propagation rather than just improving average quality. Third, does reconstruction-only training beat training with a warping loss on real blur? If so, the MRI-derived argument of Pan et al. transfers to video deblurring.
If the first check fails, we still deliver the measurement protocol and an escape-distance analysis. These would tell the field whether out-of-window matches matter at all in current recurrent deblurring.
References
- Recurrent Video Deblurring with Blur-Invariant Motion Estimation and Pixel Volumes
- ERDN: Equivalent Receptive Field Deformable Network for Video Deblurring
- Cascaded Deep Video Deblurring Using Temporal Sharpness Prior
- Motion-Compensated MR CINE Reconstruction With Reconstruction-Driven Motion Estimation
- Deep Video Deblurring for Hand-Held Cameras
- VDFlow: Joint Learning for Optical Flow and Video Deblurring
- Multi-Scale Memory-Based Video Deblurring
- Pay Attention to Hidden States for Video Deblurring: Ping-Pong Recurrent Neural Networks and Selective Non-Local Attention
- Real-World Video Deblurring: A Benchmark Dataset and an Efficient Recurrent Neural Network