Learning Attention from Attention: Efficient Self-Refinement Transformer for Face Super-Resolution

Guanxin Li, Jingang Shi, Yuan Zong, Fei Wang, Tian Wang, Yihong GongIJCAI 2023 · DOI

Abstract

Face Super-Resolution (FSR) requires the restoration of high-resolution facial details from low-resolution inputs while maintaining structural identity. Traditional attention mechanisms often suffer from blurriness due to global noise or fixed-window constraints. This paper introduces the LAA-Transformer (Learning Attention from Attention), which employs a self-refinement strategy to adaptively focus on high-frequency facial features. By integrating Region Selection Attention (RSA) and frequency-domain wavelet fusion, the model achieves state-of-the-art performance in reconstructing sharp, realistic facial textures.

Introduction

Face Super-Resolution (FSR) is a specialized task focused on restoring high-resolution (HR) faces from low-resolution (LR) inputs. Unlike general super-resolution, FSR must preserve critical facial landmarks like eyes, nose, and mouth to maintain identity. Existing attention mechanisms face a trade-off: global attention often incorporates irrelevant distant textures leading to blur, while local window attention (e.g., SwinIR) is restricted to fixed grids that may miss irregular high-frequency details in "flat" areas. The LAA-Transformer addresses this by becoming content-aware, automatically identifying and refining the most informative regions.

Methodology

The proposed Efficient Self-Refinement Transformer consists of three primary novel components designed to optimize feature extraction and reconstruction.

The architecture of the proposed FSR method, showing the feature extraction, transformation, and recovery stages.

Region Selection Attention (RSA)

RSA implements the "Learning Attention from Attention" concept through a two-step process. First, Coarse Self-Attention (CSA) is computed on downsampled features to generate a saliency map of information density. Second, the model selects the top-$k$ most critical regions for fine-grained Multi-Head Self-Attention (MHSA). This allows the model to attend to irregular, feature-rich shapes rather than being confined to square windows.

Feature Integration Module (FIM)

To move beyond treating all feature channels equally, the FIM employs a dual-branch structure. It combines channel attention with depth-wise convolutions to prioritize informative channels, enhancing the representational power of the transformer blocks.

Wavelet Fusion Module (WFM)

The WFM operates in the frequency domain using Haar Wavelets. It merges structural information (low/middle frequencies) from shallow layers with high-frequency details from deep layers. This approach ensures that the overall facial geometry is preserved while textures are realistically refined.

Experimental Results

The LAA-Transformer was evaluated on the CelebA and Helen datasets for an $8\times$ super-resolution task.

Quantitative Performance

On the Helen dataset, the model achieved a PSNR of 28.36 dB, outperforming the previous state-of-the-art Transformer (SwinIR) by 0.32 dB. It also demonstrated superior performance in SSIM and LPIPS metrics. Ablation studies confirmed that the region selection mechanism in RSA is vital, as removing it resulted in a 0.12 dB drop in PSNR.

Qualitative Analysis

Visual comparisons highlight the model's ability to produce sharper pupils and clearer nose structures with fewer artifacts compared to methods like RCAN, DIC, and SwinIR.

Qualitative comparison of the proposed method against state-of-the-art FSR models.

Conclusion

The LAA-Transformer introduces an efficient, content-aware approach to Face Super-Resolution. By leveraging self-refinement and frequency-domain fusion, it effectively captures high-frequency details without the computational overhead of global attention. Future work may address performance under extreme poses, occlusions, and scaling to even higher resolution factors.

Summary updated

BibTeX
@inproceedings{li2023learning,
  author = {Guanxin Li and Jingang Shi and Yuan Zong and Fei Wang and Tian Wang and Yihong Gong},
  title = {{Learning Attention from Attention: Efficient Self-Refinement Transformer for Face Super-Resolution}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/115},
}
Figure