TL;DR
InfoBFR is a plug-and-play framework designed to "polish" the outputs of existing blind face restoration (BFR) models. By combining a manifold information bottleneck for noise compression and a one-step diffusion LoRA for detail compensation, it effectively eliminates AI-generated artifacts—or "neural degradations"—to produce high-fidelity, realistic faces with minimal computational overhead.
What Problem Does This Paper Address?
Blind Face Restoration (BFR) aims to recover high-quality (HQ) faces from low-quality (LQ) images plagued by blur, noise, and compression. While current state-of-the-art models (like CodeFormer or GPEN) produce impressive results, they often introduce what the authors call neural degradations.
These aren't physical image defects, but rather artifacts created by the neural networks themselves:
- Topology Distortion: Weirdly shaped facial features.
- Prior Bias: The model forces a face to look like its training data rather than the original person.
- Artifact Residues: Unnatural textures or "hallucinations" in sensitive areas like teeth or eyes.
As shown in Fig. 1, existing models often struggle with these issues in complex real-world scenarios. InfoBFR acts as a universal "cleaner" to absorb the essence of these pre-trained models while discarding their mistakes.
Methods
The InfoBFR framework follows a two-step "Information Optimization" strategy:
- Information Compression (MIB): The output of a pre-trained BFR model is mapped to a latent manifold. The authors introduce a Manifold Information Bottleneck (MIB) that uses a Transformer block to gain a spatial-perception view of the face. The MIB acts as a filter, maximizing the mutual information between the restored manifold and the ground truth while minimizing the information inherited from the noisy, artifact-prone BFR output.
- Information Compensation (Diffusion LoRA): To restore the details lost during compression, the framework uses a pre-trained Stable Diffusion model. Crucially, they employ LoRA (Low-Rank Adaptation) to fine-tune the model, allowing it to synthesize high-quality facial details in just one denoising step. This avoids the massive computational cost and uncertainty of standard multi-step diffusion processes.
The entire architecture is summarized in Fig. 4, highlighting its plug-and-play nature.
Datasets
- Training: The model was trained on the FFHQ dataset, a standard benchmark for high-quality human faces.
- Evaluation: To test real-world generalization, the authors used several challenging "in-the-wild" datasets:
- Wider-Test: Severely degraded faces from the WIDER Face dataset.
- FOS: Faces with side views or heavy occlusions.
- CelebChild: Real-world photos of children.
- WebPhoto: Low-quality faces collected from the internet.
Key Insights and Ideas
- Neural vs. Data Degradation: The core insight is that the "noise" in modern BFR isn't just in the input; it's generated by the restorer. Treating AI hallucinations as a form of degradation that can be filtered via an Information Bottleneck is a novel shift in perspective.
- The "Essence" Filter: The MIB uses an information filter ($\lambda$) to decide how much of the original BFR output to keep. If $\lambda$ is too low, the result is random noise; if it's too high, the neural artifacts remain. Finding the sweet spot allows for controllable restoration.
- Efficiency via One-Step Diffusion: By starting the diffusion process from a point already close to the "finish line" (the BFR model's output), the authors prove that a single denoising step is sufficient for high-quality detail compensation.
Key Results
InfoBFR demonstrates significant quantitative and qualitative improvements across the board:
- Performance Boost: The framework achieved a "BFR-boosting" effect in nearly 85% of evaluation events. As seen in Fig. 3, it significantly improved FID (Frechet Inception Distance) and MUSIQ (image quality) scores for a wide variety of base models.
- Speed and Scale: Despite using a diffusion backbone, InfoBFR runs in approximately 70ms and uses only 16.4M trainable parameters, making it much lighter than standalone diffusion restorers.
- Visual Fidelity: Qualitative results in Fig. 6 show that InfoBFR produces cleaner teeth, more natural eyes, and better-preserved identities compared to original SOTA models.
Limitations and Future Work
While highly effective, InfoBFR is not a magic bullet. The authors note that if the underlying BFR model produces extremely severe neural degradations (where the face is essentially unrecognizable), InfoBFR may still struggle to recover the correct structure. Future work could involve tighter integration between the base model and the bottleneck to prevent these extreme failures earlier in the pipeline. Additionally, while 70ms is fast, it is still slower than pure GAN-based methods, leaving room for further optimization for real-time applications.
Summary updated