Kalman-Inspired Feature Propagation for Video Face Super-Resolution

Ruicheng Feng, Chongyi Li, Chen Change Loy

Abstract

Despite the promising progress of face image super-resolution, video face super-resolution remains relatively under-explored. Existing approaches either adapt general video super-resolution networks to face datasets or apply established face image super-resolution models independently on individual video frames. These paradigms encounter challenges either in reconstructing facial details or maintaining temporal consistency. To address these issues, we introduce a novel framework called Kalman-inspired Feature Propagation (KEEP), designed to maintain a stable face prior over time. The Kalman filtering principles offer our method a recurrent ability to use the information from previously restored frames to guide and regulate the restoration process of the current frame. Extensive experiments demonstrate the effectiveness of our method in capturing facial details consistently across video frames.

Introduction

Face Super-Resolution (FSR)—the task of reconstructing high-quality facial images from low-resolution or degraded inputs—is a classic ill-posed problem. Because significant data is lost during degradation, models must "hallucinate" missing details such as skin texture, eyelid shapes, or dental structures.

Transitioning from static images to video (VFSR) introduces the critical challenge of temporal inconsistency. Frame-by-frame restoration often leads to "identity jitter" or flickering, where facial features morph subtly between frames due to inconsistent hallucinations. Conversely, general video super-resolution models often prioritize temporal smoothness at the expense of specific facial priors, resulting in blurry or "mushy" facial reconstructions. The KEEP framework addresses this by integrating the mathematical rigor of Kalman filtering to maintain a stable, high-quality face prior across time.

Methodology

The Kalman Analogy

KEEP (Kalman-inspired fEaturE Propagation) treats video restoration as a recursive estimation problem. In this framework, the high-quality latent representation of a face is treated as a "hidden state" that is updated over time.

  1. State Prediction: The model utilizes optical flow to warp restored features from the previous frame ($t-1$) to the current frame ($t$), generating a "prior estimate."
  2. State Update: The model extracts a "measurement" from the current degraded frame. A Kalman Gain Network (KGN) then learns to compute weights that decide, on a pixel-by-pixel basis, the optimal balance between the predicted prior and the new measurement.
Detailed architecture of the KEEP framework, highlighting the Kalman Filter Network and Cross-Frame Attention.

Latent Space Propagation and Attention

Rather than operating in raw pixel space, KEEP performs propagation within the latent space of CodeFormer, a discrete codebook-based face restoration model. This approach enhances robustness against noise and alignment errors. To further stabilize local textures like hair or skin pores, the authors incorporate Cross-Frame Attention (CFA) in the decoder, allowing the model to attend to features in previous frames for enhanced consistency.

Anchoring Identity

To mitigate "identity drift"—where a subject's appearance gradually changes over a long sequence—the framework utilizes the first frame as an "anchor" within the KGN. This ensures the restoration remains locked to the subject's original identity throughout the video.

Experiments and Results

Datasets and Evaluation

The model was trained and evaluated primarily on the VFHQ (Video Face High Quality) dataset, featuring over 15,000 high-quality clips. It was also tested on real-world YouTube videos to assess generalization to authentic degradations like old film grain and low-quality webcams.

Quantitative and Qualitative Performance

KEEP achieves state-of-the-art results, outperforming existing models like BasicVSR++ and CodeFormer. Key findings include:

  • Metric Superiority: A 0.8 dB improvement in PSNR over previous leading video-based methods.
  • Identity Stability: Using ArcFace identity detection, KEEP demonstrated significantly higher identity stability (IDS) compared to frame-by-frame models, which often suffer from abrupt identity jumps.
  • Visual Fidelity: Qualitatively, KEEP produces sharper, more consistent results, avoiding the common artifacts (e.g., extra teeth or warped ears) found in image-based models.
Visual comparison of KEEP against state-of-the-art methods. KEEP demonstrates superior temporal consistency and facial detail.

Conclusion and Limitations

While KEEP excels at facial restoration, it faces a specialization trade-off. Because the model is heavily optimized for faces, non-facial regions (such as logos on hats or complex backgrounds) may occasionally appear inconsistent. Future research may explore hybrid architectures that combine KEEP’s facial expertise with general video restoration techniques to ensure holistic frame stability.

Summary updated

Abstract

Despite the promising progress of face image super-resolution, video face super-resolution remains relatively under-explored. Existing approaches either adapt general video super-resolution networks to face datasets or apply established face image super-resolution models independently on individual video frames. These paradigms encounter challenges either in reconstructing facial details or maintaining temporal consistency. To address these issues, we introduce a novel framework called Kalman-inspired Feature Propagation (KEEP), designed to maintain a stable face prior over time. The Kalman filtering principles offer our method a recurrent ability to use the information from previously restored frames to guide and regulate the restoration process of the current frame. Extensive experiments demonstrate the effectiveness of our method in capturing facial details consistently across video frames. Code and video demo are available at https://jnjaby.github.io/projects/KEEP.
BibTeX
@inproceedings{feng2024kalman,
  author = {Ruicheng Feng and Chongyi Li and Chen Change Loy},
  title = {{Kalman-Inspired Feature Propagation for Video Face Super-Resolution}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-73347-5\_12},
}
Figure