Learning Spatial Attention for Face Super-Resolution

Chaofeng Chen, Dihong Gong, Hao Wang, Zhifeng Li, Kwan-Yee K. Wong

TL;DR

The authors introduce SPARNet (Spatial Attention Residual Network), a novel architecture that uses a self-learned spatial attention mechanism to recover high-quality facial details from low-resolution images. Unlike previous methods that require expensive manual labels for landmarks or face parsing, SPARNet automatically learns to focus on key facial structures, achieving state-of-the-art performance in both image quality and identity preservation.

What Problem Does This Paper Address?

Face Super-Resolution (SR), often called "face hallucination," is the task of generating a high-resolution (HR) face from a tiny, blurry low-resolution (LR) input. This is particularly challenging because facial features—like the glint in an eye or the shape of a lip—occupy a very small percentage of the total pixels but are critical for human recognition and downstream tasks like face alignment.

Traditional SR methods treat all pixels equally, which often results in blurry features. To fix this, previous "state-of-the-art" models used multi-task learning, forcing the network to predict facial landmarks or parsing maps alongside the SR task. However, this approach has two major flaws:

  1. Data Hunger: It requires massive amounts of manually labeled data for those extra tasks.
  2. The "Chicken-and-Egg" Problem: It is incredibly difficult to accurately predict landmarks from a 16x16 pixel input, meaning errors in the landmark detector often lead to artifacts in the final SR image.

Methods

The core innovation is the Face Attention Unit (FAU). Instead of relying on external labels, the authors designed a spatial attention branch that integrates directly into a residual block.

  • Spatial Attention Branch: This branch uses an "hourglass" structure to capture multi-scale information. It produces a 2D attention map (values between 0 and 1) that acts as a weight for the feature maps. This allows the network to "bootstrap" features—effectively telling the convolutional layers to pay more attention to "hard" regions (eyes, nose, mouth) and less to "easy" regions (cheeks, background).
  • Stacking for Depth: By stacking multiple FAUs, the network learns a hierarchy of attention. Shallow layers focus on fine textures like hair, while deeper layers focus on coarse structural components.
  • SPARNetHD: To scale up to 512x512 resolution, the authors extended the model into a GAN (Generative Adversarial Network) framework. They utilized multi-scale discriminators and a sophisticated loss function combining pixel-level (L1), adversarial, feature matching, and perceptual losses to ensure the results look realistic and maintain identity.

Datasets

The authors utilized several standard and high-quality datasets:

  • Training: CelebA (179K images) for the base SPARNet and FFHQ (70K high-quality images) for the HD version.
  • Testing/Evaluation:
    • Helen: Used for standard image quality and landmark detection metrics.
  • UMD-Face: Used specifically to test identity similarity (how well the SR image matches the original person's identity).
  • CelebAHQ-Test: A synthetic test set for high-resolution evaluation.
  • CelebA-TestN: A collection of 1,051 "natural" low-quality faces to test real-world generalization.

Key Insights and Ideas

  • Attention as Implicit Supervision: The most striking "aha moment" is that the spatial attention maps naturally converge on facial components without any explicit landmark labels. The network "discovers" that eyes and mouths are the hardest parts to reconstruct and focuses its parameters there.
  • Overcoming the Imbalance: The authors frame face SR as a "hard example mining" problem. In a face image, the "easy" regions (skin) dominate the "hard" regions (features). The FAU acts as a dynamic balancer, preventing the network from being overwhelmed by the simpler background pixels.
  • Generalization to Real-World Data: Because SPARNet doesn't rely on a brittle, pre-trained landmark detector, it is significantly more robust to unaligned faces and real-world noise (like old, grainy photos) compared to previous landmark-guided methods.

Key Results

SPARNet consistently outperformed existing methods across multiple dimensions:

  • Quantitative: On the Helen dataset (8x upscale), SPARNet achieved a PSNR of 26.59 and SSIM of 0.7716, beating competitors like RCAN and Wavelet-SRNet.
  • Landmark Accuracy: Even though it wasn't trained on landmarks, the SR images produced by SPARNet allowed a standard landmark detector to achieve an AUC of 58.51%, significantly higher than the 56.23% achieved by the baseline.
  • Identity Preservation: In identity similarity tests using a pre-trained SphereFace model, SPARNet scored 0.5546, proving it preserves the "soul" of the face better than other generative models.
  • Visual Quality: Qualitatively, the model produces much sharper eyes and teeth. In the SPARNetHD version, it successfully restored a 1927 photo of Marie Curie, showing impressive handling of historical, non-synthetic degradation.

Limitations and Future Work

While SPARNet is highly effective, the authors note that the GAN-based HD version (SPARNetHD) can still be sensitive to the training distribution. If a real-world image has degradations vastly different from the synthetic noise used during training, the model might produce minor artifacts.

Future work could involve exploring blind face restoration—where the blur and noise levels are completely unknown—and further optimizing the network for mobile or real-time surveillance applications where computational resources are limited.

Summary updated

Abstract

General image super-resolution techniques have difficulties in recovering detailed face structures when applying to low resolution face images. Recent deep learning based methods tailored for face images have achieved improved performance by jointly trained with additional task such as face parsing and landmark prediction. However, multi-task learning requires extra manually labeled data. Besides, most of the existing works can only generate relatively low resolution face images (e.g., 128 × 128), and their applications are therefore limited. In this paper, we introduce a novel SPatial Attention Residual Network (SPARNet) built on our newly proposed Face Attention Units (FAUs) for face super-resolution. Specifically, we introduce a spatial attention mechanism to the vanilla residual blocks. This enables the convolutional layers to adaptively bootstrap features related to the key face structures and pay less attention to those less feature-rich regions. This makes the training more effective and efficient as the key face structures only account for a very small portion of the face image. Visualization of the attention maps shows that our spatial attention network can capture the key face structures well even for very low resolution faces (e.g., 16×16). Quantitative comparisons on various kinds of metrics (including PSNR, SSIM, identity similarity, and landmark detection) demonstrate the superiority of our method over current state-of-the-arts. We further extend SPARNet with multi-scale discriminators, named as SPARNetHD, to produce high resolution results (i.e., 512×512). We show that SPARNetHD trained with synthetic data can not only produce high quality and high resolution outputs for synthetically degraded face images, but also show good generalization ability to real world low quality face images. Codes are available at https://github.com/chaofengc/Face-SPARNet.
BibTeX
@article{chen2021learning,
  author = {Chaofeng Chen and Dihong Gong and Hao Wang and Zhifeng Li and Kwan-Yee K. Wong},
  title = {{Learning Spatial Attention for Face Super-Resolution}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=9NrEqDFL9yd},
}

Loading related content...

Figure