RestoreFormer++: Towards Real-World Blind Face Restoration From Undegraded Key-Value Pairs

Zhouxia Wang, Jiawei Zhang, Tianshui Chen, Wenping Wang, Ping Luo

Blind face restoration (BFR) is a classic "needle in a haystack" problem: how do you recover a crisp, high-quality portrait from a blurry, noisy, or heavily compressed original? RestoreFormer++ offers a powerful new answer by rethinking how models use high-quality facial "priors" and how they handle the messy reality of real-world image degradation.

Addressing the Synthetic-to-Real Gap

Most current BFR methods rely on "priors"—external information like facial landmarks or features from pre-trained generative models—to fill in missing details. However, these methods often suffer from two major flaws:

  1. Ignoring Context: They often process facial components locally, missing out on the global contextual information that helps maintain a natural, coherent look across the whole face.
  2. The Synthetic Gap: Models are typically trained on cleanly degraded synthetic data. Real-world "in-the-wild" photos often have complex issues like haze, uneven lighting, and slight misalignments that synthetic training sets fail to simulate, leading to artifacts and "uncanny valley" results.

To bridge this gap, the authors introduced the Extending Degrading Model (EDM). EDM adds realistic factors like haze, uneven degradation, and random spatial shifts. The spatial shifts are particularly effective as they teach the model to be tolerant of the slight alignment errors common in real-world preprocessing.

The RestoreFormer++ Framework

The authors introduce two core architectural components to tackle these challenges:

  • Multi-Head Cross-Attention (MHCA): Instead of the self-attention used in standard Transformers, RestoreFormer++ uses cross-attention. The "queries" are derived from the degraded face, while the "keys" and "values" come from high-quality priors. This allows the model to explicitly "look up" clean details from a reference dictionary based on the corrupted input. This process happens across multiple scales to capture both coarse structure and fine texture.
  • Reconstruction-Oriented High-Quality Dictionary (ROHQD): Many previous methods used dictionaries derived from face recognition models. RestoreFormer++ instead learns a dictionary specifically for reconstruction using vector quantization (VQ) on high-quality faces. This provides a much richer set of details for the restoration task.

Key Results and Qualitative Comparison

RestoreFormer++ consistently outperforms previous state-of-the-art methods like GPEN, VQFR, and the original RestoreFormer. On the CelebA-Test dataset, it achieved the best FID (38.41) and Identity Distance (IDD) (0.5375), striking a superior balance between looking "real" and actually looking like the original person.

The model excels at removing haze and restoring natural-looking details in eyes, teeth, and hair, even under uneven lighting. In subjective user studies on real-world images, RestoreFormer++ was preferred over competitors by a wide margin (e.g., 67.82% vs. 32.18% for VQFR).

Qualitative comparison showing RestoreFormer++ producing clearer, more natural results on real-world images compared to other methods.

Limitations and Future Work

Despite its strengths, RestoreFormer++ has a few remaining hurdles. The model still struggles when faces are partially covered by large objects (e.g., a tennis racket) or in very unusual or extreme poses. The authors note these issues stem from biases in the training data (FFHQ), which consists mostly of near-frontal, unobstructed faces. Future work will focus on incorporating more diverse poses and explicitly modeling obstacles to improve general-world robustness.

This summary is based on the research paper "RestoreFormer++: Towards Real-World Blind Face Restoration from Undegraded Key-Value Pairs".

Summary updated

BibTeX
@article{wang2023restoreformer,
  author = {Zhouxia Wang and Jiawei Zhang and Tianshui Chen and Wenping Wang and Ping Luo},
  title = {{RestoreFormer++: Towards Real-World Blind Face Restoration From Undegraded Key-Value Pairs}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  doi = {10.1109/tpami.2023.3315753},
}

Loading related content...

Figure