RestoreFormer++: Towards Real-World Blind Face Restoration from Undegraded Key-Value Pairs

Zhouxia Wang, Jiawei Zhang, Tianshui Chen, Wenping Wang, Ping Luo

Blind face restoration (BFR) is a classic "needle in a haystack" problem: how do you recover a crisp, high-quality portrait from a blurry, noisy, or heavily compressed original? RestoreFormer++ offers a powerful new answer by rethinking how models use high-quality facial "priors" and how they handle the messy reality of real-world image degradation.

Addressing the Synthetic-to-Real Gap

Most current BFR methods rely on "priors"—external information like facial landmarks or features from pre-trained generative models—to fill in missing details. However, these methods often suffer from two major flaws:

  1. Ignoring Context: They often process facial components locally, missing out on the global contextual information that helps maintain a natural, coherent look across the whole face.
  2. The Synthetic Gap: Models are typically trained on cleanly degraded synthetic data. Real-world "in-the-wild" photos often have complex issues like haze, uneven lighting, and slight misalignments that synthetic training sets fail to simulate, leading to artifacts and "uncanny valley" results.

To bridge this gap, the authors introduced the Extending Degrading Model (EDM). EDM adds realistic factors like haze, uneven degradation, and random spatial shifts. The spatial shifts are particularly effective as they teach the model to be tolerant of the slight alignment errors common in real-world preprocessing.

The RestoreFormer++ Framework

The authors introduce two core architectural components to tackle these challenges:

  • Multi-Head Cross-Attention (MHCA): Instead of the self-attention used in standard Transformers, RestoreFormer++ uses cross-attention. The "queries" are derived from the degraded face, while the "keys" and "values" come from high-quality priors. This allows the model to explicitly "look up" clean details from a reference dictionary based on the corrupted input. This process happens across multiple scales to capture both coarse structure and fine texture.
  • Reconstruction-Oriented High-Quality Dictionary (ROHQD): Many previous methods used dictionaries derived from face recognition models. RestoreFormer++ instead learns a dictionary specifically for reconstruction using vector quantization (VQ) on high-quality faces. This provides a much richer set of details for the restoration task.

Key Results and Qualitative Comparison

RestoreFormer++ consistently outperforms previous state-of-the-art methods like GPEN, VQFR, and the original RestoreFormer. On the CelebA-Test dataset, it achieved the best FID (38.41) and Identity Distance (IDD) (0.5375), striking a superior balance between looking "real" and actually looking like the original person.

The model excels at removing haze and restoring natural-looking details in eyes, teeth, and hair, even under uneven lighting. In subjective user studies on real-world images, RestoreFormer++ was preferred over competitors by a wide margin (e.g., 67.82% vs. 32.18% for VQFR).

Qualitative comparison showing RestoreFormer++ producing clearer, more natural results on real-world images compared to other methods.

Limitations and Future Work

Despite its strengths, RestoreFormer++ has a few remaining hurdles. The model still struggles when faces are partially covered by large objects (e.g., a tennis racket) or in very unusual or extreme poses. The authors note these issues stem from biases in the training data (FFHQ), which consists mostly of near-frontal, unobstructed faces. Future work will focus on incorporating more diverse poses and explicitly modeling obstacles to improve general-world robustness.

This summary is based on the research paper "RestoreFormer++: Towards Real-World Blind Face Restoration from Undegraded Key-Value Pairs".

Summary updated

Abstract

Blind face restoration aims at recovering high-quality face images from those with unknown degradations. Current algorithms mainly introduce priors to complement high-quality details and achieve impressive progress. However, most of these algorithms ignore abundant contextual information in the face and its interplay with the priors, leading to sub-optimal performance. Moreover, they pay less attention to the gap between the synthetic and real-world scenarios, limiting the robustness and generalization to real-world applications. In this work, we propose RestoreFormer++, which on the one hand introduces fully-spatial attention mechanisms to model the contextual information and the interplay with the priors, and on the other hand, explores an extending degrading model to help generate more realistic degraded face images to alleviate the synthetic-to-real-world gap. Compared with current algorithms, RestoreFormer++ has several crucial benefits. First, instead of using a multi-head self-attention mechanism like the traditional visual transformer, we introduce multi-head cross-attention over multi-scale features to fully explore spatial interactions between corrupted information and high-quality priors. In this way, it can facilitate RestoreFormer++ to restore face images with higher realness and fidelity. Second, in contrast to the recognition-oriented dictionary, we learn a reconstruction-oriented dictionary as priors, which contains more diverse high-quality facial details and better accords with the restoration target. Third, we introduce an extending degrading model that contains more realistic degraded scenarios for training data synthesizing, and thus helps to enhance the robustness and generalization of our RestoreFormer++ model. Extensive experiments show that RestoreFormer++ outperforms state-of-the-art algorithms on both synthetic and real-world datasets.
BibTeX
@article{wang2023restoreformer,
  author = {Zhouxia Wang and Jiawei Zhang and Tianshui Chen and Wenping Wang and Ping Luo},
  title = {{RestoreFormer++: Towards Real-World Blind Face Restoration from Undegraded Key-Value Pairs}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  doi = {10.1109/tpami.2023.3315753},
}
Figure