Abstract
General image super-resolution techniques have difficulties in recovering detailed face structures when applying to low resolution face images. Recent deep learning based methods tailored for face images have achieved improved performance by jointly trained with additional task such as face parsing and landmark prediction. However, multi-task learning requires extra manually labeled data. Besides, most of the existing works can only generate relatively low resolution face images (e.g., $128\times128$), and their applications are therefore limited. In this paper, we introduce a novel SPatial Attention Residual Network (SPARNet) built on our newly proposed Face Attention Units (FAUs) for face super-resolution. Specifically, we introduce a spatial attention mechanism to the vanilla residual blocks. This enables the convolutional layers to adaptively bootstrap features related to the key face structures and pay less attention to those less feature-rich regions. This makes the training more effective and efficient as the key face structures only account for a very small portion of the face image. Visualization of the attention maps shows that our spatial attention network can capture the key face structures well even for very low resolution faces (e.g., $16\times16$). Quantitative comparisons on various kinds of metrics (including PSNR, SSIM, identity similarity, and landmark detection) demonstrate the superiority of our method over current state-of-the-arts. We further extend SPARNet with multi-scale discriminators, named as SPARNetHD, to produce high resolution results (i.e., $512\times512$). We show that SPARNetHD trained with synthetic data cannot only produce high quality and high resolution outputs for synthetically degraded face images, but also show good generalization ability to real world low quality face images.
Introduction
Face Super-Resolution (SR), often referred to as "face hallucination," is the task of generating a high-resolution (HR) face from a low-resolution (LR) input. This task is particularly challenging because critical facial features—such as the eyes, nose, and mouth—occupy a small fraction of the total pixels but are vital for human recognition and downstream tasks like face alignment.
Traditional SR methods often treat all pixels equally, resulting in blurry features. To address this, previous state-of-the-art models employed multi-task learning, requiring the network to predict facial landmarks or parsing maps. However, these methods are data-hungry, requiring expensive manual labels, and often suffer from a "chicken-and-egg" problem: it is difficult to accurately detect landmarks on the very low-resolution images the network is trying to enhance.
Methodology
The core innovation of this work is the SPatial Attention Residual Network (SPARNet), which utilizes a self-learned spatial attention mechanism to focus on key facial structures without external supervision.
Face Attention Unit (FAU)
Instead of relying on external labels, the authors designed the Face Attention Unit (FAU). Each FAU integrates a spatial attention branch directly into a residual block. This branch uses an "hourglass" structure to capture multi-scale information and produces a 2D attention map. This map allows the network to adaptively "bootstrap" features, directing convolutional layers to focus on feature-rich regions (like eyes and teeth) while paying less attention to simpler regions like cheeks or the background.
Network Architecture and SPARNetHD
By stacking multiple FAUs, the network learns a hierarchy of attention, with shallow layers focusing on fine textures and deeper layers on coarse structural components. To scale up to $512\times512$ resolution, the authors introduced SPARNetHD, which incorporates multi-scale discriminators within a Generative Adversarial Network (GAN) framework. This version uses a combination of pixel-level (L1), adversarial, feature matching, and perceptual losses to ensure realistic results and identity preservation.
Key Results
SPARNet consistently outperforms existing methods across quantitative and qualitative metrics. On the Helen dataset (8x upscale), SPARNet achieved a PSNR of 26.59 and SSIM of 0.7716, surpassing competitors like RCAN and Wavelet-SRNet.
Identity and Structural Accuracy
Even without explicit landmark training, SPARNet produces images that significantly improve the performance of standard landmark detectors (achieving an AUC of 58.51%). Furthermore, identity similarity tests using SphereFace yielded a score of 0.5546, proving the model's ability to preserve the unique identity of the subject better than other generative approaches.
Real-World Generalization
SPARNet demonstrates impressive robustness to real-world, non-synthetic noise. Because it does not rely on brittle pre-trained landmark detectors, it handles unaligned or grainy photos more effectively than previous methods. This is exemplified by the successful restoration of historical imagery.
Conclusion
SPARNet represents a significant step forward in face super-resolution by introducing spatial attention as a form of implicit supervision. By focusing computational resources on the most difficult and informative parts of the face, the model achieves high-fidelity results and preserves identity without the need for additional manual labeling. Future work will explore blind face restoration and optimization for real-time applications.
Summary updated