Real video is low-resolution, low-frame-rate, noisy and compressed all at once. The models that can upscale video to any spatial size and any intermediate time are trained only on clean, idealized downsampling. The models that handle real noise and codec artifacts only upscale by a fixed 4× at the native frame rate. Simply training the continuous models on corrupted inputs leaves a deeper problem in place. A continuous video model treats every detail in the input as signal, and it spreads information across many frames. On real footage, those two habits copy noise and compression blocks into every output pixel and every synthesized frame.
We propose to split the continuous space-time video representation into two bands at a frequency cutoff that moves with the requested magnification and with how badly the input is degraded. Below the cutoff, a "base" band covers the frequencies the degraded input can actually vouch for. It is fitted to the input with a fidelity loss and may freely pool information across many frames. Above the cutoff, a separate "residual" band covers detail the input cannot vouch for. It is generated from short-range context with perceptual losses and is never propagated across long time spans. The two-sentence version is this: decide, per clip and per output scale, which frequencies to reconstruct and which to synthesize, and only let the reconstructed ones travel through time. If this works, a single model can turn real phone or internet video into arbitrarily larger, smoother-motion video without amplifying artifacts.
Core Claim
The claim has three parts and is specific enough to be wrong. First, real-world degradation mostly corrupts the high end of the low-resolution spectrum while leaving coarse color and structure intact. This is the observation behind Frequency Separation for Real-World Super-Resolution, which split single images with a fixed low-pass filter, supervised the low band with pixel loss and the high band adversarially, and won AIM 2019. It follows that the trustworthy band of a real input is narrower than its sampling limit (the Nyquist frequency) and should be estimated rather than assumed.
Second, a continuous representation built from explicit sinusoids lets that boundary be placed exactly at any output scale. We use Continuous Space-Time Video Super-Resolution with 3D Fourier Fields (V³), which models video as a sum of 3D sinusoids whose phases encode motion and which renders through an analytical blur kernel. Concretely, for a requested spatial factor $s$, temporal factor $r$, and a per-clip degradation estimate $d$, the base band keeps only sinusoids with spatial frequency below $c_x = \kappa_x(d)/s$ and temporal frequency below $c_t = \kappa_t(d)/r$. Here frequencies are in units of the output sampling rate, and $\kappa \in (0,1]$ is the fraction of the input's band that is predicted to be reliable. The $1/s$ form comes from the fact that the input's Nyquist limit sits at $1/s$ of the output's.
Third, the "propagation paradox" reported by Investigating Tradeoffs in Real-World Video Super-Resolution (RealBasicVSR) is a high-band phenomenon. That paper found that long temporal propagation accumulates noise and codec artifacts and fixed it with a separate cleaning pre-stage. We hypothesize that long-range aggregation is safe, and even beneficial, when restricted to the base band.
The resulting model is trained with
$$\mathcal{L} = \lVert P(F_{\le c}) - \mathrm{LP}_{\kappa}(x) \rVert_1 + \lambda_{\mathrm{p}}\,\mathcal{L}_{\mathrm{perc}}(F_{\le c} + R_{> c},\, y) + \lambda_{\mathrm{a}}\,\mathcal{L}_{\mathrm{adv}}(R_{> c}),$$
where the terms are defined as follows. $F_{\le c}$ is the rendered base field and $R_{>c}$ the rendered residual. $P$ is the known blur-and-sample operator back to input resolution. $\mathrm{LP}_\kappa(x)$ is the degraded input $x$ low-passed at its trusted cutoff, and $y$ is the clean high-resolution target (available on synthetic training data). The adversarial term sees only the residual band. In plain terms, the base band must match the trustworthy part of the input, while the residual band only has to look realistic.
Research Program
The method roles of prior work are as follows. V³ supplies the base representation, because its explicit global frequencies make a scale-indexed cutoff a masking operation rather than a learned approximation. Its own limitations motivate the residual band: it is soft at 16× because of L1 regression and a finite set of frequencies, and it is degradation-agnostic by the authors' own statement. VideoINR is the alternative continuous backbone. It decodes features warped by predicted optical flow directly to RGB with a coordinate network and has no frequency separation, so we use it to test whether the idea transfers beyond Fourier fields. The degradation process comes from Expanding Synthetic Real-World Degradations for Blind Video Super Resolution (SRWD-VSR), whose shuffled bucket includes realistic blur kernels, sensor-space noise, pixel binning and real codecs. We extend it with frame dropping applied before compression, so temporal decimation interacts with the codec as it does in real capture.
Four components are built. The first is a small degradation encoder that predicts $\kappa_x, \kappa_t$ from the clip. It is supervised on synthetic data by the cutoff where the degraded and clean low-resolution spectra diverge beyond a threshold. The second is the masked Fourier base field, with a long temporal window. The third is a residual head with a short temporal window, for which we compare a lightweight adversarial decoder against a one-step generative refiner. The fourth is a joint renderer that applies V³'s blur kernel to both bands so that anti-aliasing holds at any scale.
The long-tailed view of Peng Hu et al. motivates the residual band's separate loss. That work treats rare sharp structures as underweighted when full images are regressed. The multi-frame averaging base of Bordone Molini et al. is the closest precedent for letting many frames inform only the base, though that work used fixed scale and no frequency cutoff.
Two known risks constrain the design. Open problem D1.1 in the residual-learning literature warns that a deterministic base computed from a degraded input carries the degradation into the residual target. Our base is low-passed at a trusted cutoff for exactly that reason, and the residual is trained against clean targets, never against input minus base. Open problem D1.3 warns that at 8–16× the base constrains very little and the residual becomes free hallucination. We do not deny this. We measure it through temporal consistency of the residual band, because invented texture that flickers between frames is the observable symptom.
Baselines
The naive reference point is a two-stage pipeline built from simple tools. It applies classical bicubic spatial upscaling and linear frame blending, with no learning. It sets the floor for every metric and shows how much the tasks actually demand.
The strongest existing continuous method is V³ retrained on our degradation bucket with its original architecture and loss. It shares our training data, iteration count, crop sizes and a matched hyperparameter search of equal trials. This is the real competitor, because it isolates whether anything beyond "train on realistic data" is needed. For reference, V³ reports outperforming prior continuous methods by over 2 dB PSNR at 4× spatial and 8× temporal on Adobe240 V³. We found no absolute PSNR value on Lacuna for V³ on Adobe240, so we will reproduce its clean number before retraining. Beating it means higher LPIPS-measured perceptual quality and lower flicker on real-degraded evaluation at equal or better PSNR, with the gap widening at out-of-distribution scales.
VideoINR retrained on the same bucket is the second continuous baseline, under the same protocol. VideoINR reports gains of over 1–2 dB PSNR over TMNet and Zooming Slow-Mo at 16× spatial and 4× temporal VideoINR.
The strongest real-world baseline is a cascade. RealBasicVSR (pretrained, then fine-tuned on our bucket) produces cleaned ×4 video, and clean V³ performs the remaining space-time upscaling. The real-world bar on VideoLQ is set by SRWD-VSR, which reports a 7.1% NRQM improvement over RealBasicVSR. We found no absolute NRQM value on Lacuna for either method, so both will be rerun under our evaluation code.
Three ablations isolate the key ingredient:
- a fixed, scale-independent cutoff in the style of Fritsche et al. but in our continuous model;
- a cutoff scaled by $1/s$ but with $\kappa$ fixed at 1, so it is scale-aware but not degradation-aware;
- full-band long-range propagation, which removes the base-only propagation rule.
Why not just…?
Why not just train V³ or VideoINR on realistic degradations? This is the main threat, and it may close much of the gap. The argument against it is mechanistic. A single regression loss over all frequencies has no way to tell sensor noise above the trusted band from texture. RealBasicVSR found that propagation amplifies such errors even when the model is trained on degraded data, and V³'s global, multi-frame field is a long-range propagator by design. The retrained V³ baseline settles this directly. If it matches our model on real-degraded RealVSR and on the propagation-length curve described below, the split is unnecessary and we will say so.
Why not just clean first with RealBasicVSR, then run a continuous model? The cascade inherits RealBasicVSR's fixed 4× grid. Its cleaning stage also removes high-band content before the continuous model sees it, which caps the detail available for synthesis. Stage-wise error propagation is exactly what one-stage space-time methods such as Zooming Slow-Mo were built to avoid. Still, cleaning is strong and cheap, so the cascade is run as a full baseline. If it wins at 4×, our claim narrows to non-integer and out-of-distribution scales.
Why not just use a fixed frequency split, as ESRGAN-FS does? A fixed filter in output pixels means a different fraction of the input's band at every scale. At 2× it discards trustworthy detail, and at 16× it admits untrustworthy detail. The fixed-cutoff and $\kappa$-fixed ablations measure precisely this. Our prediction is that the fixed split matches us near the training scale and loses at out-of-distribution scales and on heavily compressed clips.
Why not just use a real-world video diffusion upscaler such as Upscale-A-Video, followed by frame interpolation? Such models produce strong textures at a fixed scale. They do not offer continuous spatial or temporal querying, and they reintroduce a two-stage pipeline. We include one as a perceptual-quality reference at 4× spatial, 2× temporal, without claiming to beat it on no-reference realism there.
Proposed Experiments
Training uses REDS, Vimeo-90K and Adobe240 high-frame-rate clips, degraded by the extended SRWD bucket. Training spans spatial scales from 1× to 4× and temporal factors from 2× to 8×.
Synthetic evaluation uses Adobe240 and Vid4 under held-out degradation compositions. It covers in-distribution scales and out-of-distribution ones (up to 16× spatial and 4× temporal), and reports PSNR, SSIM, LPIPS and a warping-based flicker score.
Real-world full-reference evaluation uses RealVSR. Its high-resolution frames come from a real camera rig, so we drop alternate input frames and score the synthesized in-between high-resolution frames against true captured frames. This makes it the decisive test of joint space-time restoration under real degradation. Wild evaluation uses VideoLQ with NRQM, NIQE and BRISQUE plus flicker, and K|Lens provides a check against real optical blur.
The first decisive check is the propagation-length curve. We vary the number of frames feeding the base and residual bands separately, measure the error in the high band against clean targets, and expect error to fall with more frames in the base band but rise in the residual band. Reproducing RealBasicVSR's paradox only in the high band would confirm its frequency origin.
The second decisive check correlates the predicted $\kappa$ with the measured codec quantization level and noise strength. A degradation-aware cutoff that does not track the actual degradation would mean the gains come from elsewhere.
The third decisive check is the comparison against retrained V³ on RealVSR held-out frames at non-integer scales. If it shows no meaningful margin, the central claim fails.
References
- Frequency Separation for Real-World Super-Resolution
- Continuous Space-Time Video Super-Resolution with 3D Fourier Fields
- Investigating Tradeoffs in Real-World Video Super-Resolution
- VideoINR: Learning Video Implicit Neural Representation for Continuous Space-Time Super-Resolution
- Expanding Synthetic Real-World Degradations for Blind Video Super Resolution
- Rethinking Image Super Resolution from Long-Tailed Distribution Learning Perspective
- Deepsum++: Non-Local Deep Neural Network for Super-Resolution of Unregistered Multitemporal Images
- Temporal Modulation Network for Controllable Space-Time Video Super-Resolution
- Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution
- Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolution