TL;DR
This paper introduces NASFE, a novel network architecture search (NAS) based approach for enhancing face images degraded by multiple factors like blur, noise, and low-light conditions simultaneously. By leveraging clean reference images to preserve identity and employing a NAS-designed fusion network, NASFE significantly outperforms existing methods in both quantitative metrics and visual quality.
What Problem Does This Paper Address?
In today's digital world, especially with the rise of video conferencing, we frequently encounter face images of poor quality. These degradations, stemming from factors like poor lighting, camera noise, and motion blur, significantly hinder the performance of crucial applications like face analysis, recognition, and human-computer interaction. While existing methods often focus on addressing a single type of degradation, real-world scenarios frequently involve a combination of these issues. This paper tackles the challenging problem of restoring face images degraded by multiple factors (blur, noise, and low-light) using a single, unified network.
Methods
The core of NASFE lies in its multi-task approach and its innovative use of Network Architecture Search (NAS). The network comprises three specialized encoders: one for deblurring (EB), one for denoising (EN), and one for low-light enhancement (EL). These encoders extract task-specific features from the degraded input image.
A key innovation is the fusion network, which intelligently combines these task-specific features. Instead of relying on standard convolutional blocks, the authors employ NAS to automatically design an efficient fusion architecture. This NAS-based fusion network is built using "fusion cells," which are directed acyclic graphs. To further enhance the fusion process, they introduce two novel operators:
- FFT-op: Inspired by Wiener deconvolution, this operator helps fuse features and mitigate blur effects.
- Deveiling operator: This operator is designed to effectively enhance low-light conditions.
Additionally, NASFE incorporates a classification network to identify the specific degradations present in the input image, providing a prior for the fusion process. Crucially, to preserve the subject's identity, the network utilizes clean reference images of the same person. These images provide identity information, which is then processed using VGGFace features and an Adaptive-Instance Normalization (AdaIN) layer to reduce style variations. A novel identity loss (Liden) is introduced to ensure the identity is maintained in the restored image. The overall loss function combines L2 loss, perceptual loss, and the identity loss.
Datasets
The authors trained their model using clean images from the CelebA [42] and VGGFace2 [4] datasets. They generated synthetic degraded images by applying blur kernels (from motion and Gaussian kernels), low-light factors, and realistic noise to these clean images.
For evaluation, they created several test datasets:
- Test-N: For noise.
- Test-B: For blur.
- Test-L: For low-light.
- Test-BN: For blur and noise.
- Test-BNL: For blur, noise, and low-light.
They also collected a real-world face image dataset with multiple degradations for qualitative evaluation.
Key Insights and Ideas
- Identity as a Prior: Using clean reference images to extract identity information as a prior for restoration is a powerful idea. This is more robust than relying on facial landmarks or semantic maps extracted directly from degraded images.
- NAS for Fusion: Applying NAS to design the fusion network is a smart way to optimize the combination of multi-task features, leading to better performance than hand-crafted fusion modules.
- Specialized Operators: The introduction of FFT-op and deveiling operators specifically tailored to address blur and low-light conditions, respectively, demonstrates a nuanced approach to multi-degradation restoration.
- Unified Multi-Task Restoration: The ability of a single network to effectively handle multiple, diverse degradations simultaneously is a significant step forward from single-task restoration models.
Key Results
NASFE demonstrated superior performance across various scenarios:
- Single Degradation: Outperformed SOTA denoising methods by 0.6dB PSNR and 0.02 SSIM. Outperformed SOTA deblurring methods by 1.93dB PSNR and 0.04 SSIM. Outperformed SOTA low-light enhancement methods by 3dB PSNR and 0.10 SSIM.
- Multiple Degradations (Test-BNL): Achieved a 2.1dB PSNR and 0.07 SSIM improvement over the second-best method.
- Face Recognition: Showed a 7% improvement in Top-1 accuracy on Test-BNL compared to the second-best method, highlighting the effectiveness of the restored images for downstream tasks.
- Qualitative Results: Visual comparisons showed NASFE producing significantly clearer and sharper face images with enhanced lighting, free from artifacts present in outputs from other methods.
Limitations and Future Work
- Training Data Requirement: The method requires clean reference images of the same person during training to extract identity information. While this information is not needed during inference, it's a constraint for training.
- Computational Cost: While NAS aims for efficiency, the search process itself can be computationally intensive. The paper mentions using a progressive search strategy to mitigate this.
The authors suggest that their NAS-based approach for designing the fusion network is a promising direction for future work in multi-task image restoration. The success of NASFE in handling multiple degradations also opens avenues for extending this framework to even more complex degradation scenarios.
Summary updated