Self-Enhanced Convolutional Network for Facial Video Hallucination

Chaowei Fang, Guanbin Li, Xiaoguang Han, Yizhou Yu

This academic paper, published in IEEE Transactions on Image Processing, introduces SECNet, a novel deep learning framework designed to "hallucinate" high-resolution facial details in low-quality video sequences.

TL;DR

SECNet is a self-enhanced convolutional network that achieves state-of-the-art facial video super-resolution by combining a local sliding-window approach with a recurrent refinement loop. By leveraging both previous super-resolved frames and high-level feature encoding via ConvLSTM, the model ensures that fine details like teeth and eyes remain sharp and temporally consistent across a video.

What Problem Does This Paper Address?

Facial video hallucination (or face super-resolution) is the task of generating high-resolution (HR) video from low-resolution (LR) inputs. While static image super-resolution has seen massive gains, video remains a "notorious ill-posed problem."

The primary challenges are:

  1. Temporal Consistency: Simply applying image-based methods frame-by-frame leads to "flickering" or jitter.
  2. Alignment: Faces move, rotate, and change expression. Traditional methods struggle to accurately register (align) frames over long intervals.
  3. Complex Structures: General video super-resolution (VSR) methods often fail to recover intricate, domain-specific facial features like the glint in an eye or the texture of teeth, resulting in blurry or "uncanny" results.

Methods

The authors propose a two-stage architecture called the Self-Enhanced Convolutional Network (SECNet):

  1. Local Frame Fusion Network (LFFNet): This module takes a small window of adjacent LR frames, aligns them using optical flow, and produces an initial HR estimate. It is built on a Residual Dense Network (RDN) architecture to maximize feature extraction from the local temporal neighborhood.
  2. Enhanced Recurrent Frame Fusion Module (ERFFNet): This is the core innovation. It refines the initial estimate by looking backward. It takes the super-resolved results of previous frames and feeds them back into the current frame's processing.
  3. Sequential Feature Encoding (SFE): To handle long-term dependencies, the model uses ConvLSTM units. Instead of just looking at pixels, it encodes temporal information in the "feature space." The authors experimented with three strategies: one-way, cascaded bidirectional, and fused bidirectional ConvLSTMs, finding that the bidirectional approaches significantly improved stability.
  4. Self-learned Spatial Attention: The network includes an attention mechanism within its residual blocks. This allows the model to "focus" its computational power on structurally complex areas (like eyes and mouths) while ignoring smoother regions like cheeks.

Datasets

The model was trained and evaluated on a mix of domain-specific and general datasets:

  • VoxCeleb: A massive dataset of over 140,000 sequences of tracked faces. This served as the primary training ground.
  • RAVDESS: Used for testing, this dataset contains sequences of actors speaking and singing with diverse emotional expressions.
  • Harmonic & VID4: These are general video datasets (containing buildings, animals, etc.) used to prove that SECNet isn't just a "face specialist" but also performs excellently on general super-resolution tasks.

Key Insights and Ideas

  • Recurrence is Key: The "Self-Enhanced" part of the name refers to the model's ability to use its own past outputs to improve current performance. This creates a virtuous cycle of detail recovery.
  • Feature-Level vs. Pixel-Level: While many VSR methods focus on aligning pixels, SECNet’s use of ConvLSTM to encode temporal context in high-level features makes it more robust to small alignment errors in the optical flow.
  • Attention Matters: Faces have a predictable "geography." By incorporating spatial attention, the network learns to prioritize the reconstruction of high-frequency details in regions where they are most expected (eyes, teeth, hair).

Key Results

SECNet outperformed existing state-of-the-art methods (like FRVSR and VSR-DUF) across multiple benchmarks:

  • Quantitative: On the VoxCeleb dataset, SECNet achieved a PSNR of 35.26 dB, outperforming the second-best method (FRVSR) by nearly 1 dB—a significant margin in image processing.
  • Qualitative: Visual comparisons show that SECNet recovers distinct teeth and sharp eye details where other methods produce blurry blobs.
  • Temporal Stability: The "SSIM-vt" metric (which measures temporal structure) showed that SECNet maintains much higher consistency over time, effectively eliminating the "flicker" common in other VSR models.

Limitations and Future Work

While SECNet is highly effective, the authors note a few areas for growth:

  • Facial Priors: The current model is "blind" to specific facial geometry. It doesn't explicitly use facial landmarks or 3D face models. The authors suggest that incorporating these "priors" could further improve accuracy in extreme poses.
  • Computational Cost: Recurrent models with bidirectional ConvLSTMs are computationally heavy. Future work could focus on optimizing the architecture for real-time performance on mobile devices.
  • Extreme Motion: While the model handles talking heads well, very rapid or erratic head movements still pose a challenge for the optical flow-based alignment.

Summary updated

Abstract

As a domain-specific super-resolution problem, facial image hallucination has enjoyed a series of breakthroughs thanks to the advances of deep convolutional neural networks. However, the direct migration of existing methods to video is still difficult to achieve good performance due to its lack of alignment and consistency modelling in temporal domain. Taking advantage of high inter-frame dependency in videos, we propose a self-enhanced convolutional network for facial video hallucination. It is implemented by making full usage of preceding super-resolved frames and a temporal window of adjacent low-resolution frames. Specifically, the algorithm first obtains the initial high-resolution inference of each frame by taking into consideration a sequence of consecutive low-resolution inputs through temporal consistency modelling. It further recurrently exploits the reconstructed results and intermediate features of a sequence of preceding frames to improve the initial super-resolution of the current frame by modelling the coherence of structural facial features across frames. Quantitative and qualitative evaluations demonstrate the superiority of the proposed algorithm against state-of-the-art methods. Moreover, our algorithm also achieves excellent performance in the task of general video super-resolution in a single-shot setting.
BibTeX
@article{fang2020self,
  author = {Chaowei Fang and Guanbin Li and Xiaoguang Han and Yizhou Yu},
  title = {{Self-Enhanced Convolutional Network for Facial Video Hallucination}},
  journal = {IEEE Trans. Image Process.},
  year = {2020},
  url = {https://openreview.net/forum?id=d_5YYlcN7By},
}

Loading related content...

Figure