Supervising Video Deblurring Networks with the Blur Itself

A blurry video frame is not just a damaged image. It is the average of everything the camera saw while the shutter was open. Networks that jointly remove blur and synthesize in-between frames exploit this: they read the motion hidden in the smear instead of discarding it. Yet these networks are still trained the way image classifiers were a decade ago, with one loss on the final sharp frames and nothing in between. We propose a different internal training signal. The intermediate layers of the network are asked to reproduce time-averages of the sharp video over progressively shorter sub-intervals. The shallowest layer reproduces the whole-exposure average, which is the blurry input itself. The next layer reproduces averages over half-intervals, then quarter-intervals, until the deepest layer outputs individual sharp frames. These targets can be computed exactly from the high-frame-rate footage that standard benchmarks already contain, so they cost nothing extra to obtain.

The bet is that this "temporal integral pyramid" gives early layers a target they can actually reach. A coarse time-average is a physically real, easier version of the final answer, not a premature demand for sharp texture. It should therefore help where current joint models fail most: long smears whose trajectories exceed a layer's view, and high interpolation ratios where errors compound. The pyramid also gives a diagnostic for free. When a reconstruction is wrong, the per-level errors show whether the network misplaced motion coarsely or only failed on fine sub-frame timing.

Core Claim

Joint deblurring and frame interpolation, which we call unified temporal reconstruction, has moved steadily away from cascades. BIN replaced deblur-then-interpolate pipelines with a recurrent pyramid that reads trajectories from the smear and beat the prior joint method on Adobe240 by 3.27 dB (Blurry Video Frame Interpolation). DeMFI added flow-guided feature aggregation and a recurrent refinement stage for arbitrary-time queries (DeMFI). VIDUE handled unknown exposure times with exposure-adaptive convolutions (Joint Video Multi-Frame Interpolation and Deblurring under Unknown Exposure Time). These papers motivate the proposal: each shows that keeping the smear's temporal information beats discarding it. Each is also trained with a loss only at the output, or, in Argaw et al., with intermediate supervision on optical flow borrowed from a frozen external estimator (Motion-blurred Video Interpolation and Extrapolation).

Deep supervision means attaching extra losses to intermediate layers. It has worked in neighbouring settings, but only along the spatial axis. MIMO-UNet supervises downsampled deblurred outputs at 1/4, 1/2 and full resolution in a single U-Net (Rethinking Coarse-to-Fine Approach in Single Image Deblurring). MetaNeRV attaches low-resolution prediction heads to a video representation network (MetaNeRV). These papers supply the mechanism we reuse.

Deep Trajectory Supervision constrains the design (Deep Trajectory Supervision: Deep Supervision Strikes Back). It shows that forcing early layers to match final targets fights the way networks naturally accumulate evidence with depth. Its remedy is a hand-designed schedule of softened targets. Our claim is that in video restoration the right softened targets are not heuristic: physics hands them to us. Let $S(t)$ be the latent sharp video over the normalized window $t\in[0,1]$ spanning the input exposures and the gaps between them. Partition this window into $2^k$ equal sub-intervals $I_{k,j}$ at level $k$. The level-$k$ target for sub-interval $j$ is

$$ A_{k,j} = \frac{1}{|I_{k,j}|}\int_{I_{k,j}} S(t)\,dt . $$

Level 0 over each exposure window is the observed blurry frame. The finest level $L$ is the set of sharp frames. Every intermediate target is a real, partially blurred image, consistent with every other level: averaging two adjacent level-$(k{+}1)$ targets gives the level-$k$ target.

The novelty is supervision along the temporal axis, organized by the physics of exposure. No paper in the file supervises intermediate layers of a joint deblur/interpolation network at all, beyond flow. No paper builds intermediate targets as temporal sub-integrals.

Research Program

We attach a lightweight head to each decoder stage of an existing joint backbone. The head at stage $k$ predicts $\hat A_{k,j}$ for all $2^k$ sub-intervals. Shallow stages handle coarse temporal levels and deep stages handle fine ones. The training loss is

$$ \mathcal{L} = \sum_{k=0}^{L} \lambda_k \sum_{j=1}^{2^k} \lVert \hat A_{k,j} - A_{k,j} \rVert_1 , $$

where $\lambda_k$ weights level $k$. All heads are dropped at inference, so test-time cost does not change.

Two design questions are part of the study. The first is whether temporal level should be coupled with spatial scale. A coarse time-average is smooth along the motion direction, so pairing it with a coarser spatial resolution is natural. We test both a temporal-only pyramid at full resolution and a coupled pyramid. The second is how to set $\lambda_k$ without the manual tuning that the multi-scale literature complains about. We compare a uniform weighting against a weighting proportional to measured per-level difficulty early in training.

Targets need sharp frames at the capture rate. GoPro and Adobe240 synthesize blur by averaging 7 to 11 frames from 240 FPS footage, so sub-interval averages are exact up to frame quantization. We cap $L$ so that the finest level contains at least one sharp frame.

The analysis component asks whether the hypothesis behind the method is true. We train a terminal-only backbone, freeze it, and fit linear read-outs from each decoder stage to the temporal-integral targets and, separately, to downsampled sharp targets. This is analogous to the layer-wise read-outs in Deep Trajectory Supervision. If intermediate features naturally encode time-averages better than sharp downsampled frames, the pyramid matches the network's own maturation schedule, and that explains why it should help. If they do not, the method can still help, but by imposing structure rather than by matching it, and we report that honestly.

We also use per-level errors at test time as an error-attribution tool. The goal is to separate coarse motion misplacement from fine sub-frame timing errors in failures. This addresses the credit-assignment problem that unified models inherit from cascades, where it is hard to tell whether a final artifact came from spatial deblurring or from temporal placement.

Baselines

The naive reference is a deblur-then-interpolate cascade: MIMO-UNet++ followed by an off-the-shelf interpolator, the same pairing that VIDUE shows produces ghosting. It is the reference point and not a serious contender. Argaw et al. report beating the SRN+DAIN cascade by over 2.2 dB on GoPro at 32.202 dB PSNR (Motion-blurred Video Interpolation and Extrapolation). We include the cascade mainly to calibrate per-motion-bin error profiles.

The strongest existing method is VIDUE, which the file reports as about 1.3 dB above DeMFI on GoPro ×8 and more than 1.5 dB above prior methods at ×16 (Joint Video Multi-Frame Interpolation and Deblurring under Unknown Exposure Time). We found no absolute VIDUE PSNR reported on Lacuna. On Adobe240 the anchors in the file are BIN4 at 32.59 dB, and DeMFI at about 1 dB above ALANET (DeMFI). VIDUE is the strongest competitor because it is already a unified, exposure-aware model trained with only a terminal L1 loss. That is exactly the setting where our change is isolated. We retrain VIDUE from its released configuration on identical data splits, with the same iteration count and the same learning-rate search budget, and use it as the backbone for every supervised variant. We repeat the main comparison on DeMFI to show the effect is not specific to one architecture. Beating VIDUE means a gain at ×8 and ×16 on GoPro and on Adobe240 that exceeds seed-to-seed spread over three seeds. We also require no loss on the small-motion bins.

The key ablation isolates the temporal hierarchy. On the same backbone with the same heads, we compare spatial hard targets in the style of MIMO-UNet, meaning downsampled sharp frames at each stage with its frequency-domain loss. We also compare the same spatial targets softened with a Deep Trajectory Supervision-style schedule. Finally, we compare our temporal-integral targets. Only the target definition changes across these three.

Why not just…?

Why not just add MIMO-UNet-style multi-scale losses to VIDUE? This is the obvious move, and it is a real threat. Spatially downsampled sharp frames already give early layers easier targets. However, a downsampled sharp frame still demands that the layer resolve when each pixel is. That is the hard part when smears exceed the receptive field, which Learning to Extract Flawless Slow Motion From Blurry Videos identifies as the source of ghosting. Our target-only ablation settles it directly. If spatial targets match temporal ones overall and in the large-motion bins, the temporal story is wrong, and we would report spatial deep supervision as the finding instead.

Why not just add a re-blur consistency loss at the output, requiring predicted sharp frames to average back to the input? This loss uses the same physics with no intermediate heads, and it is the most dangerous alternative. It constrains only the output, though. It sends no direct gradient to shallow decoder stages, and it checks only the full-exposure average, not the intermediate sub-intervals. We run it as a baseline both alone and combined with the pyramid. If it alone recovers most of the gain, the contribution reduces to "use reconstruction consistency," and we would say so.

Why not just use Deep Trajectory Supervision's soft schedule as is? That schedule was derived for classification, where softening means lowering confidence. In restoration there is no obvious analogue: blurring a target spatially is not the same as a less confident prediction. The softened-spatial ablation tests whether a generic schedule suffices, or whether temporal integrals are the softening that fits this task.

Why not just train longer or use a bigger backbone? Deep supervision is often a convergence aid. We therefore report full PSNR-versus-iteration curves, and we give the terminal-only baseline 1.5× the iterations and a width-matched version that includes the head parameters. If our final advantage disappears under extra training, we report only a speed-of-convergence result.

Proposed Experiments

Training and evaluation use GoPro at ×8 and ×16 and Adobe240, with cross-dataset transfer to YouTube240. For YouTube240, ALANET reports 35.89 dB PSNR when trained on Adobe240 (ALANET). Metrics are PSNR and SSIM. We also report results binned by motion magnitude, which we estimate from smear length in the ground-truth high-frame-rate footage.

For temporal stability we borrow the stabilized PSNR and its temporal-consistency variant PSNRσ from Benchmarking Video Frame Interpolation. We additionally average its 9-frame sequences into blurry inputs to obtain controlled motion-magnitude and occlusion breakdowns. That benchmark's warning that non-linear motion rewards plausible hallucination also applies here, so we report double-occlusion regions separately.

The decisive checks are three. First, the temporal-integral pyramid must beat both spatial-target ablations and the output-only re-blur loss on the same backbone. The gap should grow with motion magnitude and interpolation ratio. Second, the linear-probe study should show whether terminal-only networks already encode time-averages at intermediate depth. Third, on DeMFI's recurrent refinement stage, the per-level errors should reveal whether late-frame failures begin as coarse misplacement in early refinement passes. This is the internal error drift that unified recurrent models are suspected of. If the pyramid wins only on small motion, or only through faster convergence, we will report the narrower result rather than the broader claim.