Finding Latched Failures in Recurrent Video Restoration

Many video upscalers and learned video codecs are recurrent. They carry a hidden memory from frame to frame, so a brief event in the input can leave a trace that outlives the event. This note proposes a testing method that searches for short, natural input events, such as a half-second occlusion, a camera flash, a burst of compression blocking, or a dropped frame, that push the model into a bad internal state. The defect we look for is one that persists after the input has returned to normal. Because every frame after the event is identical to the clean video, any difference in output from that point on comes from the model's memory alone. This gives a test oracle that does not depend on trusting a perceptual quality score.

The second half of the idea is a falsifiable prediction from theory. On the choice of Perception Loss Function for Learned Video Compression shows that decoders forced to make whole sequences look realistic are penalized for correcting an early error ("permanence of error"), while decoders trained for pixel accuracy (mean squared error, MSE) or for per-frame realism can drift back. If that analysis carries over to real recurrent upscalers, then how long a latched failure lasts should grow as a model is pushed toward realism along the perception-distortion tradeoff, the proven tension between pixel accuracy and looking like a natural image (The Perception-Distortion Tradeoff). The note proposes measuring that relationship directly and checking it with human judgments. This matters because the field's standard metrics are known to misjudge exactly the outputs in question.

Core Claim

Recurrent video restoration models have reachable "latched" failure states. These are triggered by brief, realistic input events and persist for many frames after the trigger ends. A search guided by how far the model's hidden state stays from its clean-run trajectory finds such states with fewer model evaluations than random or output-guided search. Their persistence is ordered by the model's position on the perception-distortion curve: MSE-trained models such as BasicVSR should self-correct quickly, and adversarially trained, temporally regularized models such as TecoGAN should lock in.

The core measurement is a persistence length. Let $x$ be a clean clip and $x'$ the same clip with a trigger applied only on frames $t_0$ through $t_1$. Let $y_t, y'_t$ be the model's outputs on the two clips, $g_t$ the ground-truth frame, $d$ a defect measure, and $\delta$ a threshold. Persistence is

$$L(x') = \max \{\tau : d(y'_{t_1+\tau}, g_{t_1+\tau}) - d(y_{t_1+\tau}, g_{t_1+\tau}) > \delta\}.$$

$L$ counts how many frames after the trigger ends the model is still measurably worse than it would have been without the trigger. Subtracting the clean-run score removes the model's ordinary error. This matters for perceptual models: they may produce texture that differs from ground truth on every frame, and only the extra defect caused by state is counted.

Relation to Prior Work

The closest method is DeepCruiser. It fuzzed a recurrent speech recognizer by projecting hidden states onto a principal-component grid and keeping audio mutations that reached new grid cells. It raised basic state coverage from 5.5% to 28.4% and induced up to 21.5% average word error rate. We borrow its central bet, that guidance by internal state beats guidance by output. We change three things. The guidance signal is continuous divergence from the clean trajectory, which avoids the grid granularity problem the file flags for continuous hidden spaces. The target defect is persistence after the trigger rather than an error during it. And the oracle is perceptual rather than a discrete transcription error.

State-aware protocol and kernel fuzzers such as AFLNet and StateFuzz motivate the general claim that inputs which change hidden state without changing visible output are what unlock deep bugs. We use them as motivation only, not as a technique.

Two works supply the theory and the phenomenon. On the choice of Perception Loss Function for Learned Video Compression proves permanence of error and demonstrates it qualitatively on Moving MNIST, KTH and UVG. However, it studies errors that arise naturally on test distributions, never searches for the inputs that cause them, and reports no persistence measurements on video restoration. TecoGAN names "temporal drifting" in adversarial recurrent generators and proposes the temporal metrics tOF and tLP, but only as training regularizers and passive benchmarks. Our proposal turns both observations into a search problem and a quantitative test.

Three works constrain the oracle design. Pixel metrics prefer blur over slightly shifted sharp detail (Bayesian Tensor Inference for Sketch-Based Facial Photo Hallucination reports RMSE 44.90 for sharper faces versus 44.24 for a smoother baseline). Deep feature statistics miss gross structural defects (Human Preference Score). And undefended neural quality models can be pushed to SROCC of 0.30–0.57 under white-box attack on KADID-10k (BiRQA, which raises this to 0.60–0.84 with anchored adversarial training). A search that maximizes a fragile neural metric would find metric exploits rather than video failures. Two choices follow from this. We use the differential, clean-run-subtracted oracle. We also use BiRQA, an image-only metric (as its authors note in the paper), framewise as a robust perceptual defect measure, and we never use it alone as the search objective.

Research Program

Triggers come from a small parameterized family of natural events: an occluding patch with position, size and duration; a global brightness flash; a burst of motion blur; a burst of JPEG or MPEG-style blocking; a dropped or duplicated frame; and a brief hard cut to unrelated content. Each trigger lasts at most a few frames. All parameters are differentiable or cheap to sample.

The search runs against a recurrent model and keeps an archive of triggers. Its fitness is the hidden-state divergence $\|h'_{t_1+k} - h_{t_1+k}\|$, normalized per layer, measured $k$ frames after the trigger ends, plus a novelty bonus that diversifies the archive. The novelty bonus is computed by clustering the direction of the divergence vector. Candidates with high divergence are then scored with the expensive oracle, $L$ under several choices of $d$: PSNR difference, framewise BiRQA difference, and tLP difference.

Targets span the perception-distortion curve. At the distortion end sit FRVSR, BasicVSR and BasicVSR++, all MSE-type recurrent models. At the perception end sits TecoGAN. For a controlled sweep, we retrain one recurrent architecture at five adversarial-loss weights in the style of SRGAN, producing a curve within a single model family. A learned video codec trained with per-frame realism versus joint-sequence realism losses, following Salehkalaibar et al., gives the cleanest test of the theory.

We will also run a human study on a few hundred found failures. Raters see the post-trigger frames of the clean and triggered runs side by side and judge which, if either, is defective. The results establish which oracle choice of $d$ matches people, and how often PSNR flags harmless texture differences or misses structural ones in perceptual models.

Baselines

The naive reference point is the clean benchmark itself. We report tOF, tLP and PSNR on REDS4, Vid4 and UVG with no triggers, plus random triggers drawn uniformly from the same family at the same evaluation budget. If random sampling already finds long-persistence failures as often as the guided search, state guidance adds nothing. The comparison counts persistence-$L$ failures above a human-validated threshold per thousand model evaluations.

Two baselines isolate the key ingredient, state guidance. The first, output-guided search, uses the identical archive and trigger family but scores candidates by framewise output defect during and just after the trigger, with no hidden-state access. The second is a DeepCruiser-style grid-coverage search, which projects hidden states onto principal components and rewards new cells, as in DeepCruiser. Its implementation is re-tuned for video features. The proposal beats these if it finds more distinct long-persistence failures per budget, with human-confirmed defects, across at least three of the four MSE-type and perceptual targets.

The strongest competitor is white-box gradient attack through time. It runs projected gradient ascent on the same trigger parameters, maximizing excess defect summed over the $m$ frames after the trigger, backpropagating through the recurrence. It gets the same number of forward-backward evaluations, counted as forward-equivalents, and the same hyperparameter search budget. This is the real threat, because it directly optimizes what we measure. For reference, no reported result found on Lacuna for adversarial or fuzzing attacks on any recurrent video super-resolution model. The nearest reported numbers are the DeepCruiser coverage and word-error figures above, and BiRQA's attacked SROCC of 0.60–0.84 on KADID-10k. We cannot inherit a target, so all baselines are rerun.

A sliding-window model, EDVR (which BasicVSR reports BasicVSR exceeds by 0.33 dB on REDS4), serves as a negative control. By construction its persistence cannot exceed its window length. Any longer $L$ there means the oracle is broken.

Why not just…?

Why not just run per-frame adversarial attacks or random noise? Recurrent models with flow-guided alignment are built to average away isolated frame corruption, as the accumulation behavior in FRVSR and BasicVSR++ suggests. Per-frame attacks also measure damage during the attack, not after it. The random-trigger baseline answers this directly. If it matches guided search on persistence, this objection wins.

Why not just backpropagate through time with a persistence loss? This may well work as well or better on differentiable models, and we treat it as the main baseline rather than a straw man. The search keeps two practical advantages. It needs no gradients, so it applies to codecs with non-differentiable entropy coding and to deployed systems. Its novelty archive returns diverse failure modes rather than one optimum. If gradient attack matches our method on differentiable targets, we will say so. The contribution then narrows to the persistence oracle, the gradient-free setting, and the perception-distortion finding, which stands regardless of which search finds the triggers.

Why not just report tOF and tLP on the standard benchmarks? Those metrics average over clips with no adversarial events, so they cannot tell a model that self-corrects in five frames from one that latches for fifty after a rare event. They also rely on optical flow, which TecoGAN's own setting makes fragile under occlusion. The human study will show whether framewise robust scoring with clean-run subtraction agrees with raters better than tLP does.

Why not just train with per-frame realism or use sliding windows, avoiding latching altogether? Salehkalaibar et al. report that per-frame realism trades lock-in for flicker. BasicVSR++ reports a 1.3 dB PSNR gain over sliding-window EDVR with 65% fewer parameters, so recurrence will stay in use. Our controlled sweep and codec comparison measure the cost directly: they show how much persistence each training choice removes and how much flicker (tLP) it adds.

Decisive Checks

Three results would settle the proposal. The central result is a monotone relationship in the controlled adversarial-weight sweep: median persistence of found failures rising with adversarial weight while clean PSNR falls. Holding the architecture fixed is what makes this a test of the tradeoff rather than of model family. The joint-realism codec should show longer $L$ than the per-frame-realism codec under identical triggers, extending Salehkalaibar et al.'s qualitative observation to a measured, searched-for worst case. Finally, on the human-labeled set, differential BiRQA should agree with raters more often than differential PSNR on perceptual models, while the two agree on MSE models. That would show the objective-subjective gap is itself state-dependent. A flat persistence curve, or an EDVR control that latches, would falsify the claim.