Consider turning a blurry, low-frame-rate video into a sharp, high-frame-rate one. This is useful for handheld phone footage, dashcams, and sports clips. For real footage, the training pairs such a model needs do not exist. No one can film the same moment both blurry and sharp. Today's best joint models therefore train on synthetic pairs: they average consecutive frames of a high-speed clip to fake motion blur, then learn to undo it. That shortcut breaks on real cameras, because real blur passes through a nonlinear camera pipeline, sensor noise, and compression after the light is integrated. We propose a way to use real, unlabeled footage as a training signal. Hide one real blurry frame. Ask the model to produce the sharp frames that span that frame's exposure, using only the frames before and after it. Then re-blur the model's own output through a small learned camera model and require it to reproduce the hidden frame.
The key observation is that half of the synthetic blur recipe is physically correct and half is not. A blurry frame really is an average of the sharp scene over the exposure interval, but only in linear light, before the camera's tone curve, noise, and compression are applied. We keep the correct half fixed as physics and learn the incorrect half per video. We use the frames on both sides of the hidden frame as the constraint that prevents the model from cheating. The result is a sim-to-real adaptation signal that needs no paired data and no hand-built degradation simulator.
Core Claim
A joint deblur-and-interpolate network trained on synthetic averaged blur can be adapted to real video by self-supervision on withheld real frames. The loss has three parts: exposure integration in linear radiance, a per-clip learned camera response, and prediction conditioned on both the past and future neighbors. We claim this adaptation improves real-world output more than the obvious fix of training on richer synthetic degradations, and that the two are complementary. Concretely, let $B_k$ be a real observed blurry frame that we withhold from the input. Let $f_\theta(\tau \mid B_{k-1}, B_{k+1})$ be the network's sharp linear-radiance frame at time $\tau$, predicted from the two neighbors. Let $E_k$ be the set of sampled times inside frame $k$'s exposure window, whose length is a per-clip duty-cycle parameter estimated jointly. Let $g_\psi$ be a small camera model (tone curve plus noise and compression level) fitted per clip. The real-data loss is
$$ \mathcal{L}_{\text{real}} = \Big\| \, B_k - g_\psi\Big( \tfrac{1}{|E_k|} \textstyle\sum_{\tau \in E_k} f_\theta(\tau \mid B_{k-1}, B_{k+1}) \Big) \Big\|_1 , $$
and it is added to the usual supervised loss on synthetic pairs. The supervised term anchors what "sharp" looks like. The real term forces the sharp trajectory to explain real observations it never saw as input.
Relation to Prior Work
The bidirectional mechanism comes from imputation work showing that conditioning on both sides of a gap beats forward-only rollout. BRITS runs forward and backward recurrent networks with a consistency penalty and reaches imputation MAE 0.278 on PhysioNet 2012, against 0.445 for M-RNN. Graph Imputer anchors off-screen football trajectories at disappearance and reappearance. LoCI-DiffCom conditions a 3D diffusion model on preceding and subsequent MRI scans. All three use bidirectional conditioning as the model's inference structure on in-domain data. We use it differently: as a source of supervision on out-of-domain real video. The withheld frame serves as a target, not a missing value to be filled.
The closest existing model is Blurry Video Frame Interpolation. Its BIN architecture already re-blurs interpolated sharp frames to reconstruct blurry inputs as a cycle loss and reports 32.59 dB PSNR on Adobe240. BIN supplies our backbone, and its limitation motivates the proposal. Its cycle loss is computed on synthetic data, in sRGB, on frames the network also received as input. That setup cannot reveal a mismatch between synthetic and real blur formation, and it is satisfiable by near-identity solutions. Our loss differs in three ways: it is computed on real clips, on a frame withheld from the input, and after a learned camera model applied to linear-light averages. Temporal Cycle-Consistency Learning likewise extracts supervision from temporal structure without labels, but it learns alignment embeddings and reconstructs no pixels.
The case that simple synthetic degradation fails on real sensors comes from single-image work. Real-ESRGAN improves NIQE on RealSR-Canon from 6.77 to 4.53 by cascading blur, noise, resizing, and JPEG. Deblurring by Realistic Blurring learns realistic blur from unpaired photos and reaches 31.10 dB on GOPRO, against 30.10 dB for SRN. S2R-Removal shows the same lesson for video reflections, at 28.84 dB versus 27.04 dB on S2R-Ref. These works constrain our design, and each is purely spatial or clip-level, with no use of held-out frames. They improve the simulator. We instead learn the non-physical part of the degradation from the real clip itself.
Unsupervised Non-Rigid Image Distortion Removal via Grid Deformation is the nearest precedent for fitting a per-sequence model to real footage without pairs. It assumes a static scene and needs 65 to 499 seconds per clip. We borrow its per-clip fitting idea only for the small camera model $g_\psi$ and keep the restoration network amortized across videos. PixelDA adapts synthetic images to real appearance at the pixel level, but it is limited to photometric shifts. Motion blur is a geometric, time-integrated shift, which is exactly what our exposure-integration term models.
Research Program
The backbone is BIN, modified to output linear-radiance frames at arbitrary intermediate times so that exposure integration can be computed. The first stage trains on Adobe240, synthesizing blur by averaging frames in linear light rather than sRGB. This alone tests whether the placement of the linearization matters.
The second stage mixes synthetic batches with real clips. Real clips come from YouTube240, which is in-the-wild high-speed footage that we subsample to low frame rate. We also collect our own handheld phone videos at native frame rates. On each real clip we withhold frame $k$, predict from $k-1$ and $k+1$, and apply $\mathcal{L}_{\text{real}}$. The camera model $g_\psi$ is deliberately low-capacity: a monotone tone curve, a scalar noise level, and a JPEG quality estimate. It is too weak to absorb scene content, so the burden of explaining the frame stays on $f_\theta$.
Two failure modes need direct analysis. The first is the degenerate solution in which $f_\theta$ outputs already-blurry frames. Blurring a blurry frame again should not match $B_k$ unless the exposure is estimated as near zero. We therefore monitor the estimated duty cycle and the sharpness of $f_\theta$'s outputs throughout training. The second is non-identifiability between exposure length and motion magnitude. We test for it on synthetic clips, where both are known, by checking whether the estimated duty cycle recovers the true one.
Baselines
The naive reference is BIN trained only on Adobe240 with the standard sRGB averaging recipe, evaluated directly on real footage. It establishes the size of the gap we claim to close. The published number is 32.59 dB PSNR on Adobe240 (BIN4, Blurry Video Frame Interpolation). We found no reported result on Lacuna for BIN, or any joint method, on real footage such as YouTube240; the paper evaluates it only qualitatively.
The strongest competitor is BIN trained on a richer simulator, because this is what a practitioner would try first. For this baseline we apply Real-ESRGAN's second-order degradation cascade after the averaging step. In a variant, we replace averaging with blur generated by a BGAN-style learned blur model in the manner of Deblurring by Realistic Blurring. The comparison is fair if both arms see the same synthetic source clips, the same number of gradient steps, and the same hyperparameter search budget. Our arm additionally sees the unlabeled real clips, so we also give the simulator arm those clips through a PixelDA-style appearance translation that uses them as target domain. Our method counts as beating this baseline if it wins on the real-reference benchmark described below by a margin larger than seed variance. Our method stacked on the richer simulator should beat either alone if the two are complementary.
The cascade baseline runs a single-image real-world deblurrer (DBGAN) and then a frame interpolator, matched in parameter count as closely as available models allow. BIN reports that joint modeling beats the EDVR+DAIN cascade by 3.27 dB on Adobe240. We include the cascade anyway because the ranking could change on real data.
Four ablations isolate the ingredients:
- No withholding. The loss is applied only to frames the network also received as input. This is essentially BIN's cycle loss moved to real data.
- No camera model. $g_\psi$ is removed, so averaging happens in sRGB.
- Forward-only conditioning. The withheld frame is predicted from $B_{k-2}, B_{k-1}$ instead of $B_{k-1}, B_{k+1}$.
- Test-time only. The real-data loss is applied per clip at inference instead of during training, in the style of the grid-deformation method, with runtime reported.
Why not just…?
Why not just train BIN on Real-ESRGAN-style high-order degradations? This is the real threat. If a hand-built cascade already covers the distribution of real phone blur, our adaptation adds little. The published evidence cannot settle this. Real-ESRGAN's gains are for single-image super-resolution and are measured by NIQE, and no video interpolation result under such degradations appears in the file. The richer-simulator baseline and the stacked combination above decide it. If the simulator alone matches our method on the real-reference benchmark, we would report that and reposition the method as a way to diagnose simulator gaps rather than close them.
Why not just cascade a real-world deblurrer with an off-the-shelf interpolator? BIN's 3.27 dB advantage over EDVR+DAIN suggests that cascades propagate deblurring artifacts into flow estimation. More fundamentally, a cascade cannot use our loss, because the interpolator never outputs the sub-exposure sharp frames whose average must match the hidden frame. We still run it on real data, since the synthetic ranking may not transfer.
Why not just adapt appearance with PixelDA or CycleGAN? PixelDA changes color, illumination, and noise statistics while preserving content. Real-versus-synthetic motion blur also differs in how exposure time relates to trajectory, which a per-image translator has no access to. The PixelDA-augmented simulator arm tests this directly. If it closes most of the gap, the exposure-integration term is not earning its place.
Why not just use forward-only prediction of the hidden frame? Evidence from BRITS and the football imputer shows that one-sided context drifts. For our purpose there is a sharper reason: a forward-only target lets the model explain the frame by extrapolating motion that may be wrong, and nothing on the far side checks it. The forward-only ablation measures how much the second anchor contributes.
Proposed Experiments
The central difficulty is evaluating on real footage without real pairs. We address it three ways.
First, we build a small real-reference set, targeting on the order of thirty clips. A beam splitter feeds two synchronized cameras: one at long exposure and low frame rate, and one at short exposure and high frame rate that serves as sharp ground truth. We report PSNR and SSIM on the interpolated sharp frames. This set is the decisive benchmark.
Second, we use held-out real frames from YouTube240 and our phone videos in observation space. Here we predict a withheld frame with the camera model frozen, from a model that never trained on that clip, and report PSNR against the real frame. This is informative but partly circular, so it is secondary.
Third, we report no-reference NIQE on the output frames, following the Real-ESRGAN protocol, and a paired user preference study on in-the-wild clips, in the manner of the S2R-Real evaluation.
On synthetic data we report Adobe240 PSNR against the 32.59 dB BIN4 figure, and GOPRO PSNR against the 31.10 dB DBGAN+ figure. These checks confirm that real-data adaptation does not erode synthetic performance by more than a small margin.
The decisive checks are three. On the real-reference set, our method should beat both the naive and the richer-simulator baselines. The no-withholding ablation should lose most of that gain, showing that the held-out target, not merely real data, is what matters. On synthetic clips with known exposure, the estimated duty cycle should track the true one, showing that the camera model is not absorbing motion. If the no-withholding ablation matches the full method, the core claim is wrong.
References
- BRITS: Bidirectional Recurrent Imputation for Time Series
- Multiagent off-screen behavior prediction in football
- LoCI-DiffCom: Longitudinal Consistency-Informed Diffusion Model for 3D Infant Brain Image Completion
- Blurry Video Frame Interpolation
- Temporal Cycle-Consistency Learning
- Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data
- Deblurring by Realistic Blurring
- From Synthesis to Removal: Physics-Grounded Reflection Simulation and Diffusion-Based Video Dereflection
- Unsupervised Non-Rigid Image Distortion Removal via Grid Deformation
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks