Abstract
As a domain-specific super-resolution problem, facial image hallucination has enjoyed a series of breakthroughs thanks to the advances of deep convolutional neural networks. However, the direct migration of existing methods to video is still difficult to achieve good performance due to its lack of alignment and consistency modelling in temporal domain. Taking advantage of high inter-frame dependency in videos, we propose a self-enhanced convolutional network for facial video hallucination. It is implemented by making full usage of preceding super-resolved frames and a temporal window of adjacent low-resolution frames. Specifically, the algorithm first obtains the initial high-resolution inference of each frame by taking into consideration a sequence of consecutive low-resolution inputs through temporal consistency modelling. It further recurrently exploits the reconstructed results and intermediate features of a sequence of preceding frames to improve the initial super-resolution of the current frame by modelling the coherence of structural facial features across frames. Quantitative and qualitative evaluations demonstrate the superiority of the proposed algorithm against state-of-the-art methods. Moreover, our algorithm also achieves excellent performance in the task of general video super-resolution in a single-shot setting.
Introduction
Facial video hallucination, or face super-resolution (SR), involves generating high-resolution (HR) video from low-resolution (LR) inputs. While static image super-resolution has seen massive gains, video remains a notoriously ill-posed problem. The primary challenges in this domain are maintaining temporal consistency to avoid "flickering," ensuring accurate alignment as faces move and rotate, and recovering complex, domain-specific structures like the glint in an eye or the texture of teeth. Traditional general-purpose video super-resolution (VSR) methods often fail to capture these intricate details, resulting in blurry or uncanny reconstructions.
Methodology
The proposed Self-Enhanced Convolutional Network (SECNet) addresses these challenges through a two-stage architecture designed to maximize both local temporal information and long-term dependencies.
Local Frame Fusion Network (LFFNet)
The first stage utilizes a small window of adjacent LR frames. These frames are aligned using optical flow and processed through a Residual Dense Network (RDN) architecture. This module produces an initial HR estimate by maximizing feature extraction from the local temporal neighborhood.
Enhanced Recurrent Frame Fusion Module (ERFFNet)
The core innovation of SECNet is its recurrent refinement loop. This module takes the super-resolved results of previous frames and feeds them back into the current frame's processing. This "self-enhancement" allows the model to use its own past high-quality outputs to improve current performance, creating a virtuous cycle of detail recovery.
Sequential Feature Encoding (SFE)
To handle long-term dependencies and ensure temporal stability, SECNet employs ConvLSTM units. By encoding temporal information in the feature space rather than just the pixel space, the model becomes more robust to minor alignment errors. The authors found that bidirectional ConvLSTM approaches significantly improved stability over one-way models.
Self-learned Spatial Attention
Faces have a predictable geography. SECNet incorporates a spatial attention mechanism within its residual blocks, allowing the network to prioritize computational resources on structurally complex areas like the eyes and mouth while ignoring smoother regions like the cheeks.
Experiments and Results
Datasets
The model was trained primarily on VoxCeleb, a dataset containing over 140,000 sequences of tracked faces. Evaluation was conducted on RAVDESS (actors with diverse emotional expressions) and general video datasets like Harmonic and VID4 to demonstrate the model's versatility beyond facial domain-specific tasks.
Quantitative and Qualitative Performance
SECNet outperformed existing state-of-the-art methods, such as FRVSR and VSR-DUF, across multiple benchmarks. On the VoxCeleb dataset, SECNet achieved a PSNR of 35.26 dB, outperforming the second-best method (FRVSR) by nearly 1 dB.
Visually, the model excels at recovering distinct teeth and sharp eye details where other methods produce blurry artifacts. Furthermore, the "SSIM-vt" metric, which measures temporal structure, confirmed that SECNet maintains higher consistency over time, effectively eliminating the jitter common in other VSR models.
Discussion and Future Work
SECNet represents a significant step forward in facial video hallucination by combining local fusion with recurrent refinement. However, the authors identify several areas for future research. Currently, the model does not explicitly utilize facial landmarks or 3D geometry priors; incorporating such domain knowledge could improve accuracy in extreme poses. Additionally, the computational cost of bidirectional ConvLSTMs remains high, suggesting a need for architectural optimization for real-time applications. Finally, while the model handles standard speech well, extremely rapid or erratic head movements still present challenges for the underlying optical flow-based alignment.
Summary updated