DiffMAC: Diffusion Manifold Hallucination Correction for High Generalization Blind Face Restoration

Nan Gao, Jia Li, Huaibo Huang, Zhi Zeng, Ke Shang, Shuwu Zhang, Ran HearXiv 2024 · DOI · arXiv

Introduction

Blind Face Restoration (BFR) aims to recover high-quality facial images from inputs suffering from unknown, complex degradations such as blur, noise, and compression. While modern GAN and Transformer-based models produce sharp results, they often suffer from two major flaws: low generalization to non-photorealistic domains (e.g., paintings, statues) and the generation of "hallucination" artifacts—incorrect details like double eyelashes or distorted textures that destroy the subject's identity.

Comparison of BFR methods across photorealistic and heterogeneous domains. DiffMAC (green) achieves superior performance in both categories compared to existing methods.

The DiffMAC Framework

To address these challenges, the authors propose DiffMAC, a two-stage Diffusion-Information-Diffusion (DID) framework. The core innovation is the Manifold Information Bottleneck (MIB), which acts as a sophisticated filter between two diffusion stages to ensure both quality and fidelity.

Two-Stage Restoration Process

  1. Stage I (Alignment): A latent diffusion model aligns the restored face with the spatial features of the low-quality (LQ) input using Adaptive Instance Normalization (AdaIN). This removes primary degradations but can still produce uncontrollable artifacts.
  2. Manifold Information Bottleneck (MIB): This is the framework's central mechanism. It uses information theory to compress the image "manifold" (the mathematical representation of the face), maximizing the preservation of structural identity while minimizing the influence of noise and restoration-irrelevant details. This prevents the model from hallucinating features that weren't in the original.
  3. Stage II (Refinement): The model uses the compressed manifold and identity injection (via ArcFace embeddings) to fine-tune the results. This ensures the output is high-resolution and faithful to the original person's identity and the specific artistic style of the scene.

Experimental Results

The model was trained on the standard FFHQ dataset and evaluated across diverse benchmarks, including synthetic (CelebA-HQ), real-world (LFW-Test, CelebChild), and a new Heterogeneous Face in the Wild (HFW) dataset containing 1,200 images of sculptures, oil paintings, and anime characters.

Qualitative comparison of restoration quality. DiffMAC consistently ranks highest in both photorealistic and heterogeneous (non-human style) restoration tasks.

Key Performance Metrics

  • Quantitative Superiority: On the LFW-Test dataset, DiffMAC achieved a state-of-the-art FID score of 37.75, significantly outperforming GAN-based methods like CodeFormer (52.02) and other diffusion models like DiffBIR (38.97).
  • Heterogeneous Generalization: In subjective user studies on the HFW dataset, DiffMAC scored 4.4 for style preservation, nearly doubling the performance of CodeFormer (2.5). While other models often add "human" eyes to statues or messy textures to paintings, DiffMAC maintains the original medium's texture while sharpening facial features.
  • Identity Preservation: By filtering noise through the MIB, the model avoids common generative artifacts, ensuring that the restored face remains recognizable as the original subject.

Limitations and Future Work

Despite its performance, DiffMAC's two-stage diffusion process is computationally expensive. It takes approximately 10.78 seconds to process a single image on an NVIDIA RTX 3090, which is considerably slower than GAN-based alternatives. Additionally, the model can still struggle with extremely degraded contours, such as ambiguous jawlines. Future research will explore efficient distillation techniques to accelerate inference for real-time applications.

Summary updated

Abstract

Blind face restoration (BFR) is a highly challenging problem due to the uncertainty of degradation patterns. Current methods have low generalization across photorealistic and heterogeneous domains. In this paper, we propose a Diffusion-Information-Diffusion (DID) framework to tackle diffusion manifold hallucination correction (DiffMAC), which achieves high-generalization face restoration in diverse degraded scenes and heterogeneous domains. Specifically, the first diffusion stage aligns the restored face with spatial feature embedding of the low-quality face based on AdaIN, which synthesizes degradation-removal results but with uncontrollable artifacts for some hard cases. Based on Stage I, Stage II considers information compression using manifold information bottleneck (MIB) and finetunes the first diffusion model to improve facial fidelity. DiffMAC effectively fights against blind degradation patterns and synthesizes high-quality faces with attribute and identity consistencies. Experimental results demonstrate the superiority of DiffMAC over state-of-the-art methods, with a high degree of generalization in real-world and heterogeneous settings. The source code and models will be public.
BibTeX
@misc{gao2024diffmac,
  author = {Nan Gao and Jia Li and Huaibo Huang and Zhi Zeng and Ke Shang and Shuwu Zhang and Ran He},
  title = {{DiffMAC: Diffusion Manifold Hallucination Correction for High Generalization Blind Face Restoration}},
  year = {2024},
  eprint = {2403.10098},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.10098},
}
Figure