Live video is increasingly rewritten by neural networks at the receiver. Super-resolution models upscale compressed streams, and generative decoders rebuild whole frames from a few kilobits. These systems fail in two different ways over time. Sometimes the output flickers: details change from frame to frame even when the scene does not. Sometimes the output makes a mistake early and then keeps it, such as a wrong tone, an invented texture, or a hallucinated object. Each later frame looks perfectly realistic on its own. Current quality monitors handle neither case well. Per-frame quality scores cannot see a locked-in error, because every frame is individually plausible. Clip-level video quality models buffer seconds of future frames, so they cannot run causally. Their temporal features, which compare adjacent frames, also confuse a decoder that is correcting itself with one that is flickering.
We propose a monitor that runs frame by frame, sees only the past, and uses a signal the receiver already has: the low-resolution, compressed input that the enhancer was given. The method has three steps. First, compare a perceptually blurred, downsampled copy of each output frame against the input. Second, split the discrepancy into a slow part that persists across frames and a fast part that does not. Third, measure flicker only relative to how much the input itself changed. The resulting monitor reports two interpretable numbers per frame: locked-in error and flicker. It needs no future frames, no human labels at run time, and almost no compute. The scientific claim is that this decomposition captures human judgments of enhanced streams that existing no-reference and clip-based metrics systematically get wrong.
Core Claim
The temporal objective–subjective gap in enhanced video is not one gap but two, and they pull in opposite directions. The theory of perceptual video compression in On the Choice of Perception Loss Function for Learned Video Compression shows this precisely. That paper compares decoders that enforce realism of whole frame sequences with decoders that enforce realism of each frame separately. Sequence-level decoders lock an initial frame's error into every later frame ("permanence of error"). Frame-level decoders self-correct back toward the true content but flicker. That paper uses the two regimes to choose a training loss. It does not build anything that detects, at run time, which regime a stream is in. Our claim is that a causal monitor can make this distinction cheaply, provided it has access to some reduced reference. In receiver-side enhancement, the input stream is exactly such a reference.
Consider what happens without one. A locked-in error is a fidelity error, not a realism error, so no purely no-reference model can detect it. This is a structural limit and holds no matter how large the model is.
The Monitor
Let $x_t$ be the low-resolution frame the receiver gets at time $t$ and $y_t$ the enhanced output. Let $D$ downsample to the input resolution and $G$ be a Gaussian low-pass filter. The filter is the same contrast-sensitivity proxy that Perceptual Fidelity Aware Mean Squared Error uses to raise rank correlation with human scores above raw mean squared error on static images. The per-frame residual map is $r_t = G(D(y_t)) - G(x_t)$, computed in an opponent color space. We split it with a slow running average:
$$p_t = (1-\beta)\,p_{t-1} + \beta\, r_t, \qquad f_t = r_t - p_t,$$
where $p_t$ is the persistent residual map (locked-in error), $f_t$ the transient residual, and $\beta$ a decay rate that sets how long an error must persist to count as "locked."
Flicker is measured separately. It is the output's change in a light feature space $\phi$, minus the change the input itself justifies:
$$F_t = \max\big(0,\ \|\phi(y_t)-\phi(y_{t-1})\| - \kappa\,\|\phi(x_t)-\phi(x_{t-1})\|\big),$$
where $\kappa$ is a learned scale. Subtracting the input's change means real motion and scene cuts are not penalized. A self-correcting decoder shows a shrinking $\|p_t\|$ and is rewarded, not punished. This is the reverse of what adjacent-frame differencing does in MD-VQA.
A small regressor maps $(\|p_t\|, \|f_t\|, F_t)$, optionally together with a cheap no-reference frame score, to predicted human judgments. It is calibrated once, offline. The computation is causal by construction: it uses one stored frame plus one residual map, so memory stays bounded however long the stream runs.
Kreuz and colleagues found that causal re-engineering costs stability. In Measuring real-time synchronization in both spike trains and continuous time series they rebuilt symmetric similarity measures into strictly causal ones and observed added noise sensitivity. The choice of $\beta$ is where that cost lands for us, so we study it explicitly.
Research Program
The work has three parts.
The first part is a controlled stimulus set in which the two failure modes are known. We train sequence-level and frame-level perceptual decoders on UVG and KTH, following the setup of On the Choice of Perception Loss Function for Learned Video Compression. This yields matched streams with permanence of error and with flicker. We then add enhancement streams from StreamSR's YouTube content (Exploring Real-Time Super-Resolution: Benchmarking and Fine-Tuning for Streaming Content). On these we run frame-wise super-resolution models such as EfRLFN, and we inject controlled persistent tone shifts and texture hallucinations of graded duration. Finally, where code permits, we include causal generative-decoder outputs in the style of High-Fidelity Causal Video Diffusion Models for Real-Time Ultra-Low-Bitrate Semantic Communication. That system already transmits a downscaled pixel stream, so our reduced reference exists natively there. Its authors evaluated only offline, full-reference metrics.
The second part is a subjective study that separates the two judgments viewers make. Viewers rate stimuli in two conditions: alone, which tests realism, and beside the source, which tests fidelity. They also give continuous ratings over time, so we can measure how quickly human scores react to onset and recovery of each failure. This study is essential. No benchmark in our file isolates locked-in error, and the existing ones (TaoLive, VDPVE, StreamSR's preference study) record clip-level or pairwise scores only.
The third part is a forward test, following the logic of Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarking and the live-versus-replay split of CLEF 2017 NewsREEL Overview. We freeze the monitor and its calibration, then apply it to streams uploaded after the freeze date and enhanced live. A smaller second round of ratings tests whether calibration survives new content.
Baselines
The naive reference is per-frame PSNR between the output and the bicubically upsampled input, averaged causally. It is the cheapest causal measure available. It is known to diverge from human preference: StreamSR reports EfRLFN preferred in 77.4% of comparisons against NVIDIA VSR despite closely matched PSNR. We run it on identical stimuli. Beating it on correlation with continuous human ratings is the minimum bar.
The strongest existing clip-level competitors are MD-VQA and SB-VQA. MD-VQA reports SRCC 0.942 / PLCC 0.945 on TaoLive (MD-VQA). SB-VQA reports a main score of 0.7635 on the VDPVE test set versus Fast-VQA's 0.7330 (SB-VQA). They are the strongest because they model temporal change and enhancement artifacts explicitly. We run them with their published weights, then retrain them on our stimulus training split with the same data and an equal hyperparameter search budget. Each runs both in its native buffered mode and in a causal sliding-window mode with matched latency. We do not expect to beat MD-VQA's 0.942 on TaoLive, because TaoLive is compression-only and has no enhancement input to serve as reference. Beating them means three things on our enhanced stimuli, judged with reference visible: higher rank correlation, correct ranking of self-correcting over locked-in streams, and matched or lower latency.
The strongest no-reference image metric is BiRQA (BiRQA: Bidirectional Robust Quality Assessment for Images), applied per frame and averaged causally. BiRQA reports roughly 15 FPS at 1080p with 5.5M parameters. For a high-capacity reference point we use SR-VQA (SR-VQA) offline as an upper-bound oracle. It reports +30.59% relative SRCC on LIVE-YT-HFR, but its 7B backbone rules it out as a live monitor.
Two ablations isolate the key ingredients. The first is the reduced-reference residual with no temporal split, which is plain blurred-and-downsampled MSE accumulated over time. The second is the temporal split with no input-relative flicker correction. Each must be outperformed, or the corresponding piece is unnecessary.
Why not just…?
Why not just run a strong no-reference model per frame and smooth it over time? This handles flicker partially, but it cannot see locked-in error. Sequence-level perceptual decoders produce frames that are individually realistic by design (On the Choice of Perception Loss Function for Learned Video Compression). A model shown only the output therefore has nothing to flag. The decisive test is the reference-visible condition on permanence-of-error stimuli. We predict that smoothed BiRQA scores and SR-VQA will rate locked-in and correct streams similarly, while humans with the reference will not. If humans also fail to penalize locked errors once the reference is visible, our fidelity arm has no perceptual justification, and we will report that.
Why not just buffer two-second chunks and use MD-VQA or SB-VQA? For dashboards, a two-second delay is tolerable, and we concede that. The problem is what the temporal features measure. MD-VQA's adjacent-frame feature differencing scores a decoder snapping back to the truth as a temporal spike, and it scores a constant wrong tone as stable. Neither model sees the enhancer's input. The matched-latency comparison, and the pairwise ranking of self-correcting versus locked-in streams, will show whether this matters. If retrained chunk models rank these pairs correctly, the decomposition adds little and the contribution reduces to latency.
Why not just compute blurred-and-downsampled error against the input and average it? This is the real threat. It is essentially our persistent arm without the split or the flicker correction, and it inherits PAMSE's cheapness and alignment. It is our first ablation. Our expectation is that it under-penalizes flicker, because transient errors average out, and over-penalizes deliberate super-resolution detail, which a learned weighting can discount. If it matches the full monitor on continuous human ratings, we will say so. The contribution would then be the subjective protocol and the finding that this simple reduced reference suffices, rather than the monitor.
Proposed Experiments
The first decisive check is pairwise. On matched stream pairs where one decoder self-corrects and the other locks in an error, we measure how often each metric agrees with the reference-visible human preference. We predict that adjacent-difference and no-reference metrics sit near chance and the monitor does not.
The second check is a responsiveness curve. Using continuous ratings, we compare each metric's lag and overshoot at the onset and recovery of injected errors, across values of $\beta$. This quantifies the causal stability cost that Kreuz and colleagues described qualitatively.
The third check is clip-level competitiveness. We report rank and linear correlation on our enhanced test split for all baselines under equal tuning budgets. We also report our scores on TaoLive and VDPVE by adding the cheap no-reference branch, to show the monitor does not lose standard ground. We do not claim the reduced-reference arm applies there.
The fourth check is throughput. We report frames per second and memory at 1080p and 4K over multi-hour streams on one GPU and on CPU, next to BiRQA's reported ~15 FPS. NewsREEL's lesson was that methods missing their deadline lose to simple fallbacks.
The final check is the post-freeze forward test. Calibration fitted before the freeze must keep its correlation with new ratings within a preregistered margin. If it does not, we will report the size of the drop as the operational cost of offline calibration.
References
- On the Choice of Perception Loss Function for Learned Video Compression
- Perceptual Fidelity Aware Mean Squared Error
- MD-VQA: Multi-Dimensional Quality Assessment for UGC Live Videos
- Measuring real-time synchronization in both spike trains and continuous time series
- Exploring Real-Time Super-Resolution: Benchmarking and Fine-Tuning for Streaming Content
- High-Fidelity Causal Video Diffusion Models for Real-Time Ultra-Low-Bitrate Semantic Communication
- Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarking
- CLEF 2017 NewsREEL Overview: Offline and Online Evaluation of Stream-based News Recommender Systems
- SB-VQA: A Stack-Based Video Quality Assessment Framework for Video Enhancement
- BiRQA: Bidirectional Robust Quality Assessment for Images
- SR-VQA: Super-Resolution Video Quality Assessment Model