TL;DR
This paper tackles the challenge of restoring high-quality face images from severely degraded ones by proposing a novel approach that leverages both neural architecture search (NAS) and multiple facial priors. Their method, MFPSNet, automatically designs an optimal network architecture and effectively fuses information from diverse facial cues (like parsing maps, heatmaps, and dictionaries) to achieve state-of-the-art blind face restoration results.
What Problem Does This Paper Address?
Blind Face Restoration (BFR) aims to recover high-quality (HQ) face images from low-quality (LQ) images where the degradation process is unknown. While facial priors (like semantic maps or geometric heatmaps) have proven useful, current methods face two main hurdles: 1) designing effective network architectures often requires extensive manual tuning, and 2) efficiently combining information from multiple facial priors to capture complementary details is a complex task. This paper seeks to address these limitations by automating architecture design and developing a robust multi-prior fusion strategy for BFR.
Methods
The authors introduce two main components:
- Face Restoration Searching Network (FRSNet): This network employs Neural Architecture Search (NAS) to automatically discover an optimal upsampling architecture for feature extraction. It uses a hierarchical search space that allows for the selection of optimal upsampling paths and feature transformation operations. A key innovation here is a "degradation disentanglement strategy" that jointly trains on fully and singly degraded images to better handle complex degradations.
- Multiple Facial Prior Searching Network (MFPSNet): Building upon FRSNet, MFPSNet integrates multiple facial priors (parsing maps, facial heatmaps, and facial dictionaries) into the restoration process. It uses dedicated "prior search modules" for each type of prior, employing NAS to find optimal architectures for extracting features from them. A "feature fusion module" then intelligently combines these prior features with the image features to generate HQ images. The NAS is applied to both the prior extraction and fusion modules, allowing for a holistic optimization.
Datasets
- Training: FFHQ dataset (70,000 high-quality face images).
- Synthetic Testing: CelebA-HQ and CASIA-WebFace datasets.
- Real-World Testing: A custom dataset of 981 real-world LQ face photos.
The authors also describe a degradation model used to synthesize LQ images from HQ images for training and synthetic testing, incorporating blur, noise, downsampling, and JPEG compression.
Key Insights and Ideas
- NAS for BFR is Effective: Applying NAS to the specific problem of BFR, particularly for designing the upsampling decoder, can lead to architectures that outperform hand-designed ones.
- Multi-Prior Fusion is Crucial: Different facial priors offer complementary information (structure, shape, texture). Effectively fusing these diverse cues is key to achieving high-fidelity restoration, especially for severely degraded images.
- Degradation Disentanglement Helps: Explicitly training the network to disentangle different degradation factors (even implicitly through joint training) improves its ability to handle complex, unknown degradations.
- Prior-Specific Architectures: Not all priors are the same; different priors might benefit from different feature extraction architectures, which MFPSNet's NAS approach can discover.
- Over-reliance on Large Operation Pools can Hurt: The authors found that using a very large set of candidate operations in NAS can lead to suboptimal architectures, suggesting that careful selection is important.
Key Results
- State-of-the-Art Performance: MFPSNet achieves state-of-the-art results on both synthetic (CelebA-HQ, CASIA-WebFace) and real-world datasets, outperforming existing BFR methods in terms of PSNR, SSIM, and MS-SSIM metrics.
- Superior Qualitative Results: Qualitative comparisons show that MFPSNet generates more faithful and realistic face images, with accurate facial contours and components, even for severely degraded inputs. It avoids the artifacts and low-fidelity issues seen in some GAN-based methods.
- Effectiveness of Components: Ablation studies demonstrate the importance of the upsampling search space (FRSNet-w/o-uss), the degradation disentanglement strategy (FRSNet-w/o-Dis), and the multi-prior fusion (MFPSNet variants).
Limitations and Future Work
- Limitations of Facial Priors: The paper acknowledges that the current facial priors (parsing maps, heatmaps, dictionaries) may lack information about non-facial elements like sunglasses or complex backgrounds. This can lead to limitations in generating clear contours for these elements.
- Future Work: The authors suggest extending the approach to incorporate priors that capture more diverse information beyond just facial features, which could further improve restoration quality, especially for challenging cases. They also highlight the potential for future research in exploring more sophisticated fusion strategies and NAS search spaces.
Summary updated