When you apply a state-of-the-art image restoration model to a video frame by frame, the result is almost always a jittery, shimmering mess. A model might perfectly remove rain streaks, sharpen a face, or upscale a low-resolution image when looking at a single frame. However, when these frames are played in sequence at 30 frames per second, high-frequency visual artifacts—known as temporal flickering or inter-frame inconsistency—emerge.
Understanding why this happens and how modern architectures solve it requires looking at video restoration not as a sequence of independent image problems, but as a continuous trajectory through a latent space.
The Root Cause: The Ill-Posed Mapping Problem
To understand why frame-by-frame processing fails, we must look at the mathematical nature of image restoration. Tasks like super-resolution, deblurring, and inpainting are inherently ill-posed. This means that for a single degraded input frame, there are many equally valid, high-quality clean frames that could have generated it.
When an image-centric neural network processes Frame $t$, it selects one plausible reconstruction from this set of valid solutions. When it moves to Frame $t+1$, even if the underlying scene has barely changed, slight variations in noise, compression artifacts, or lighting cause the network to select a different valid reconstruction.
As identified by Chongyi Li et al. in Kalman-Inspired Feature Propagation for Video Face Super-Resolution, this arbitrary switching between valid interpretations manifests as distracting visual flicker, identity shifts in faces, and crawling textures. The network lacks a mechanism for temporal memory, meaning fine details are allowed to shift or disappear entirely between adjacent time steps, a problem also detailed in LVTINO: LAtent Video consisTency INverse sOlver for High Definition Video Restoration.
Frame t (Low Res) ───► [ Image Model ] ───► Plausible Clean Detail A
▲ (Inconsistent!)
Frame t+1 (Low Res) ───► [ Image Model ] ───► Plausible Clean Detail B
To eliminate flickering, a model must be constrained so that its predictions are conditioned not just on the current degraded frame, but on the historical and contextual trajectory of the video.
Architectural Memory via Recurrent Propagation
The most intuitive way to enforce temporal consistency is to give the network a memory. Instead of starting fresh with each frame, recurrent video restoration networks pass hidden features or past predictions forward through time.
In Deep Video Inpainting, Dahun Kim et al. introduced a recurrent feedback loop that allows the network to "borrow" missing content from neighboring frames. By aggregating features across multiple frames and feeding the previous frame's output back into the system, the model ensures that newly synthesized pixels match the appearance and motion of what came before. Similarly, Wenhan Yang et al. utilized a recurrent framework with dual-level flow in Frame-Consistent Recurrent Video Deraining With Dual-Level Flow to prevent the model from generating slightly different background elements across a sequence.
A more explicit way to handle this state tracking is to borrow from classical control theory. Chongyi Li et al. implemented a learned "Kalman Gain" to recurrently update latent features. By treating video face restoration as a state estimation problem, their network balances the current frame's visual evidence against a running, stable face prior, drastically reducing identity flickering.
Spatio-Temporal Receptive Fields
An alternative to passing memory forward sequentially is to expand the network's receptive field across both space and time simultaneously. By processing a block or window of frames at once, the network can directly optimize for smooth transitions.
Early approaches, such as the work by Jose Caballero et al. in Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation, relied on explicit motion compensation to align neighboring frames before feeding them into a spatio-temporal network. This ensures that the network tracks the same physical object across time, preventing localized blurring or shimmering.
Modern architectures have evolved to use implicit alignment. For instance, in TempFormer: Temporally Consistent Transformer for Video Denoising, Mingyang Song et al. designed an implicit spatio-temporal mixer that bypasses expensive optical flow computations entirely. Instead, the architecture uses attention mechanisms to look across frames, ensuring that static regions maintain absolute visual continuity.
In the domain of generative models, Yizhou Li et al. proposed a sliding window attention mechanism within a diffusion framework in TDM: Temporally-Consistent Diffusion Model for All-in-One Real-World Video Restoration, forcing the diffusion process to generate textures that remain anchored across consecutive time steps.
[ Frame t-1 ] ──┐
[ Frame t ] ──┼─► [ Spatio-Temporal Mixer / Attention ] ──► Flicker-Free Frame t
[ Frame t+1 ] ──┘
Enforcing Consistency via Relational Losses
Even with spatio-temporal architectures, models can still introduce subtle flickering if their training loss functions only evaluate individual frame quality. To fix this, researchers design explicit temporal consistency losses that penalize frame-to-frame variance.
A common pitfall of early temporal losses was their reliance on optical flow to map pixels from Frame $t$ to Frame $t+1$. If the optical flow estimation failed due to occlusions or lighting changes, the loss function would enforce incorrect consistency, introducing new artifacts.
To overcome this, Xin Yu proposed a relation-based temporal loss in Video Demoiréing with Relation-Based Temporal Consistency. Instead of forcing exact pixel alignment, this loss ensures that the relationships between different regions in a frame (such as relative color and texture structures) remain invariant across time steps.
Similarly, Mingyang Song et al. introduced an "Overlap Loss" during recurrent fine-tuning, which penalizes discrepancies in overlapping regions processed by sliding windows, ensuring smooth, jitter-free transitions across the entire video timeline.
Anchoring Reconstructions with Spatial References
Another powerful intuition in video restoration is that degradation is rarely uniform across time. In a blurry video sequence, a camera shake might ruin three frames, but the fourth frame might be perfectly sharp.
In Bringing Events into Video Deblurring with Non-consecutively Blurry Frames, Wei Shang et al. demonstrated that these sporadic sharp frames can act as "Nearest Sharp Frames" (NSFs). Instead of guessing high-frequency details out of thin air for a blurry frame, the network can look ahead or backward in time, locate a sharp reference frame, and copy the preserved textures.
Frame t (Blurry) ───┐
Frame t+1 (Blurry) ─┼─► [ Texture Propagation ] ──► Restored Frame t (Stable)
Frame t+2 (Sharp) ──┘ (Borrows high-frequency details from Frame t+2)
By anchoring the reconstruction of degraded frames to actual preserved textures elsewhere in the video, the network eliminates the guesswork that causes flickering. This concept of using external structural anchors is also seen in video compression enhancement; Xiaoyi He et al. demonstrated in Enhancing HEVC Compressed Videos with a Partition-masked Convolutional Neural Network that feeding the structural partition maps used by the video encoder directly into a CNN provides a rigid spatial reference that prevents the network from generating fluctuating, inconsistent details.
Key Foundational Work
Video Reflection Removal Through Spatio-Temporal Optimization (2017) — contributed 1 concept elements to this direction.
End-to-End United Video Dehazing and Detection (2018) — contributed 1 concept elements to this direction.
Enhancing HEVC Compressed Videos with a Partition-masked Convolutional Neural Network (2018) — contributed 1 concept elements to this direction.