Abstract
Despite the promising progress of face image super-resolution, video face super-resolution remains relatively under-explored. Existing approaches either adapt general video super-resolution networks to face datasets or apply established face image super-resolution models independently on individual video frames. These paradigms encounter challenges either in reconstructing facial details or maintaining temporal consistency. To address these issues, we introduce a novel framework called Kalman-inspired Feature Propagation (KEEP), designed to maintain a stable face prior over time. The Kalman filtering principles offer our method a recurrent ability to use the information from previously restored frames to guide and regulate the restoration process of the current frame. Extensive experiments demonstrate the effectiveness of our method in capturing facial details consistently across video frames.
Introduction
Face Super-Resolution (FSR)—the task of reconstructing high-quality facial images from low-resolution or degraded inputs—is a classic ill-posed problem. Because significant data is lost during degradation, models must "hallucinate" missing details such as skin texture, eyelid shapes, or dental structures.
Transitioning from static images to video (VFSR) introduces the critical challenge of temporal inconsistency. Frame-by-frame restoration often leads to "identity jitter" or flickering, where facial features morph subtly between frames due to inconsistent hallucinations. Conversely, general video super-resolution models often prioritize temporal smoothness at the expense of specific facial priors, resulting in blurry or "mushy" facial reconstructions. The KEEP framework addresses this by integrating the mathematical rigor of Kalman filtering to maintain a stable, high-quality face prior across time.
Methodology
The Kalman Analogy
KEEP (Kalman-inspired fEaturE Propagation) treats video restoration as a recursive estimation problem. In this framework, the high-quality latent representation of a face is treated as a "hidden state" that is updated over time.
- State Prediction: The model utilizes optical flow to warp restored features from the previous frame ($t-1$) to the current frame ($t$), generating a "prior estimate."
- State Update: The model extracts a "measurement" from the current degraded frame. A Kalman Gain Network (KGN) then learns to compute weights that decide, on a pixel-by-pixel basis, the optimal balance between the predicted prior and the new measurement.
Latent Space Propagation and Attention
Rather than operating in raw pixel space, KEEP performs propagation within the latent space of CodeFormer, a discrete codebook-based face restoration model. This approach enhances robustness against noise and alignment errors. To further stabilize local textures like hair or skin pores, the authors incorporate Cross-Frame Attention (CFA) in the decoder, allowing the model to attend to features in previous frames for enhanced consistency.
Anchoring Identity
To mitigate "identity drift"—where a subject's appearance gradually changes over a long sequence—the framework utilizes the first frame as an "anchor" within the KGN. This ensures the restoration remains locked to the subject's original identity throughout the video.
Experiments and Results
Datasets and Evaluation
The model was trained and evaluated primarily on the VFHQ (Video Face High Quality) dataset, featuring over 15,000 high-quality clips. It was also tested on real-world YouTube videos to assess generalization to authentic degradations like old film grain and low-quality webcams.
Quantitative and Qualitative Performance
KEEP achieves state-of-the-art results, outperforming existing models like BasicVSR++ and CodeFormer. Key findings include:
- Metric Superiority: A 0.8 dB improvement in PSNR over previous leading video-based methods.
- Identity Stability: Using ArcFace identity detection, KEEP demonstrated significantly higher identity stability (IDS) compared to frame-by-frame models, which often suffer from abrupt identity jumps.
- Visual Fidelity: Qualitatively, KEEP produces sharper, more consistent results, avoiding the common artifacts (e.g., extra teeth or warped ears) found in image-based models.
Conclusion and Limitations
While KEEP excels at facial restoration, it faces a specialization trade-off. Because the model is heavily optimized for faces, non-facial regions (such as logos on hats or complex backgrounds) may occasionally appear inconsistent. Future research may explore hybrid architectures that combine KEEP’s facial expertise with general video restoration techniques to ensure holistic frame stability.
Summary updated