Remember What Was Visible: Occlusion-Gated Memory for Flicker-Free Video Restoration

Video restoration models such as super-resolution or denoising networks carry information forward from frame to frame. That is how they avoid flicker, the shimmering that appears when each frame is cleaned up independently and the model invents slightly different detail each time. The carried-forward information is usually aligned with optical flow, a per-pixel estimate of where each point moved. Flow has no valid answer for a surface that is hidden behind something else. When a hand passes in front of a face, or a pedestrian crosses a storefront, the network writes occluder features into its running state. When the surface reappears, the information about how it looked has already been overwritten, so the model invents its appearance from scratch. Flicker and texture tearing therefore cluster at the moments and places where content disappears and reappears. These are exactly the regions that average metrics over whole frames hide.

We propose a restoration network that keeps a second, slowly updated memory alongside its usual running state. This memory is only written when the network believes a location is truly visible, and it is read back, at coarse resolution, only for the small set of locations the network believes are occluded or newly revealed. Everything else goes through cheap dense local attention. The bet is that the occlusion signal can do three jobs at once. It decides what may enter the memory, so occluders do not contaminate the target. It decides which tokens are allowed to spend expensive long-range sampling, so the cost stays bounded. And it decides when to retrieve old appearance instead of hallucinating new appearance, which removes flicker on re-revealed surfaces.

Core Claim

Temporal inconsistency in restored video comes mostly from disocclusion events, meaning surfaces that were hidden and then reappear. It can be removed by a memory whose write rate is set by an inferred visibility score and whose read happens at a coarse pyramid level. This works better than masking attention more carefully within the current temporal window, because the information needed after a long occlusion is no longer present in that window at all. The same visibility score lets us confine deformable, long-range sampling to a small fraction of tokens. That makes wall-clock latency, not just FLOP count, competitive with a recurrent restoration transformer that has no notion of occlusion.

Concretely, let $h_t(x)$ be the coarse-level feature the recurrent network computes at location $x$ in frame $t$, let $v_t(x) \in [0,1]$ be the predicted probability that $x$ shows the target surface rather than an occluder or an unreliable warp, and let $m_t(x)$ be the memory, stored in a coordinate frame carried along by the same coarse alignment the network already uses. The memory update is

$$m_t(x) = m_{t-1}(x) + \alpha\, v_t(x)\,\big(h_t(x) - m_{t-1}(x)\big),$$

where $\alpha$ is a small learned base rate. Occluded locations ($v \approx 0$) therefore freeze their memory instead of absorbing the occluder. This mirrors the gain in Kalman-style propagation, but the gain is tied to visibility rather than predicted from appearance. Retrieval happens only for tokens with $v_t(x) < \tau$, where $\tau$ is a threshold. For those tokens, a query cross-attends to a small set of deformably sampled memory locations, and the sampling weights are multiplied by the memory's own stored visibility, so the model never reads from slots that were themselves written while occluded.

Research Program

The base network is a recurrent multi-scale restoration transformer in the style of RVRT. RVRT processes short clips in parallel, keeps a recurrent state across clips, and aligns frames with guided deformable attention, which uses flow as a starting point and then samples a few learned offsets with content-dependent weights. RVRT supplies the method we build on. Its weakness, which motivates this work, is that its sampling has no occlusion logic: where flow is invalid, it samples occluder tokens. Following DAP, which showed that a coarse-to-fine deformable attention pyramid handles large motion cheaply in a causal setting, we place the visibility head and the memory at the coarsest level. At that level, a single token covers enough area that a short occlusion rarely hides all of a surface's neighbours. Finer levels then refine retrieved coarse features with ordinary local attention, so high-frequency texture is reconstructed from a stable coarse anchor rather than chosen afresh each frame.

The visibility head is trained with supervision from synthetic occlusion: moving occluders with known masks are pasted onto clean clips. This mirrors the synthetic-occlusion construction of YouTube-VOI used by VOIN and the sim-to-real masks of IRAIS. The head is additionally regularized by forward-backward feature consistency, so it also flags warp failures that no mask labels. At test time no mask is given. This is the main difference from the occlusion-aware inpainting line. MADIN learns offsets that avoid a known hole in a single image, and our offset constraint borrows its idea of scaling the search range with local corruption. DeViT prunes known-invalid pixels from cross-frame patch matching inside a fixed window at a single scale. Neither has memory beyond its window, and both are told where the hole is. A related retrieval idea appears in facelet compensation for talking heads, which queries a token bank built from a fixed canonical source image. Our memory is instead written online, from the video itself, by the visibility score.

The efficiency question is treated as an empirical one, because Xu et al. show that the irregular gathers in multi-scale deformable attention dominate runtime despite their small FLOP share. Low-visibility tokens are therefore compacted into a contiguous batch before deformable retrieval, and the remaining tokens run fused dense window attention. Latency should behave roughly like the dense cost plus $p$ times the deformable cost, where $p$ is the fraction of routed tokens. This is a first-order model that ignores the overhead of compaction itself. We will measure $p$ on real footage and report wall-clock time and memory on fixed hardware, not FLOPs alone.

Baselines

The do-nothing reference is the same restoration backbone applied frame by frame with no temporal state, followed by an exponential moving average over time. This shows how much flicker there is to remove and how much ghosting naive smoothing adds. As Lao et al. argue, unaligned temporal averaging superimposes moving content.

The strongest existing method is RVRT. It reports a PSNR gain of 0.36 dB over BasicVSR++ on REDS4 with 10.8M parameters and 1GB of test memory, versus 27GB for VRT. No absolute RVRT PSNR value is given in the file, so we will retrain it and report our own numbers. It is the right opponent because it already combines recurrence, multiple scales and deformable attention; any gain we show must come from occlusion handling, not from those generic ingredients. We will run it from its released configuration on identical training data, including the same synthetic-occlusion augmentation we use, so that it sees the same occlusion examples. It gets the same iteration count and the same hyperparameter search budget. Beating it means two things. First, a clear reduction in temporal error and warping error inside disoccluded regions. Second, PSNR within noise of RVRT on unoccluded REDS4 and no worse wall-clock latency.

For the online, low-compute regime, DAP is the reference. It reports 26 FPS (38 ms per frame) on an RTX 2080Ti at 330 GFLOPs, with a 0.06 dB PSNR gain over EDVR-M on REDS. We will compare against it with our model restricted to causal operation. For faces, KEEP reports a 0.8 dB PSNR gain over the next best method on VFHQ-mild, along with reduced identity fluctuation and pose jitter. Its Kalman gain is the closest existing propagation rule to ours, which makes it the natural test of whether visibility-tied gain beats appearance-predicted gain under occluders such as hands or microphones. For denoising, TempFormer is the flow-free reference. It reports lower temporal residual error than VRT on DAVIS at about 40% less inference time.

The key ablation is RVRT plus a visibility mask with no memory: the same visibility head simply zeroes the attention logits of low-visibility samples. This isolates whether the gain comes from the memory or merely from masking. Further ablations remove routing (deformable retrieval for every token), move the memory to the finest scale, and replace the learned visibility with a forward-backward flow consistency check.

Why not just…?

Why not just mask RVRT's attention weights with an occlusion estimate? This is the most serious threat, and it is the masked-RVRT ablation above. Masking stops occluders from being sampled. But once a surface has been hidden for longer than the clip plus the reach of the recurrent state, there are no valid keys left to attend to. The file names exactly this as the breakdown of borrowing mechanisms under extreme occlusion. We will bucket test events by occlusion duration. If masked RVRT matches us across all durations, the memory is unnecessary and the contribution shrinks to visibility-aware masking. We predict the two match for short occlusions and diverge as duration grows.

Why not just use full spatio-temporal attention over a long window? That would place the old appearance inside the window. However, RVRT's own comparison puts VRT-style attention at 27GB of test memory against 1GB, and cost grows quadratically with clip length. A window long enough to span multi-second occlusions is not practical. Our memory has constant size per location.

Why not just restore frames independently and smooth over time? This is the do-nothing baseline. KEEP and Lao et al. show that independently restored frames disagree structurally, and that averaging them produces double exposures rather than consistency. We expect this baseline to fail most visibly in exactly the disocclusion regions we stratify on.

Why not just use bidirectional flow propagation, so a surface hidden now is borrowed from the future? This helps offline but is unavailable in causal use. It also still relies on flow through the occluded span, which Lao et al. document as hallucinated. We will include an offline bidirectional variant of RVRT and report the causal and offline settings separately.

Proposed Experiments

Training uses REDS and Vimeo-90K for super-resolution, DAVIS for denoising at the TempFormer noise levels, and VFHQ for faces. All three are augmented with pasted moving occluders whose masks are used only to supervise the visibility head. For evaluation we use REDS4 and Vid4 unchanged to check that nothing is lost without occlusion. We also use occluded versions of REDS4, DAVIS and VFHQ, with occluders drawn from held-out object categories, and YouTube-VOI. On YouTube-VOI the comparison point is VOIN's reported 46.33 PSNR, 0.989 SSIM and 0.013 LPIPS, although that setting gives masks, so it tests the memory rather than the visibility head.

Metrics are PSNR, SSIM, LPIPS, warping error and temporal residual error. KEEP's identity and pose fluctuation measures are used on faces. Every metric is reported separately for always-visible, occluded and just-revealed pixels, and by occlusion duration. Efficiency is reported as wall-clock time per frame, peak memory and the routed fraction $p$, measured on one GPU type against RVRT and DAP.

The decisive check is the duration-bucketed comparison with masked RVRT on revealed pixels. The memory must reduce temporal error on surfaces that return after long occlusions while matching masked RVRT on short ones. A second check is that real-footage $p$ stays low enough for measured latency to be at or below RVRT's. If either check fails, the paper reports that visibility-aware masking, not memory, is what the problem needs.