Network Architecture Search for Face Enhancement

Rajeev Yasarla, Hamid Reza Vaezi Joze, Vishal M. PatelarXiv 2021 · OpenReview

TL;DR

This paper introduces NASFE, a novel network architecture search (NAS) based approach for enhancing face images degraded by multiple factors like blur, noise, and low-light conditions simultaneously. By leveraging clean reference images to preserve identity and employing a NAS-designed fusion network, NASFE significantly outperforms existing methods in both quantitative metrics and visual quality.

What Problem Does This Paper Address?

In today's digital world, especially with the rise of video conferencing, we frequently encounter face images of poor quality. These degradations, stemming from factors like poor lighting, camera noise, and motion blur, significantly hinder the performance of crucial applications like face analysis, recognition, and human-computer interaction. While existing methods often focus on addressing a single type of degradation, real-world scenarios frequently involve a combination of these issues. This paper tackles the challenging problem of restoring face images degraded by multiple factors (blur, noise, and low-light) using a single, unified network.

Methods

The core of NASFE lies in its multi-task approach and its innovative use of Network Architecture Search (NAS). The network comprises three specialized encoders: one for deblurring (EB), one for denoising (EN), and one for low-light enhancement (EL). These encoders extract task-specific features from the degraded input image.

A key innovation is the fusion network, which intelligently combines these task-specific features. Instead of relying on standard convolutional blocks, the authors employ NAS to automatically design an efficient fusion architecture. This NAS-based fusion network is built using "fusion cells," which are directed acyclic graphs. To further enhance the fusion process, they introduce two novel operators:

  • FFT-op: Inspired by Wiener deconvolution, this operator helps fuse features and mitigate blur effects.
  • Deveiling operator: This operator is designed to effectively enhance low-light conditions.

Additionally, NASFE incorporates a classification network to identify the specific degradations present in the input image, providing a prior for the fusion process. Crucially, to preserve the subject's identity, the network utilizes clean reference images of the same person. These images provide identity information, which is then processed using VGGFace features and an Adaptive-Instance Normalization (AdaIN) layer to reduce style variations. A novel identity loss (Liden) is introduced to ensure the identity is maintained in the restored image. The overall loss function combines L2 loss, perceptual loss, and the identity loss.

Datasets

The authors trained their model using clean images from the CelebA [42] and VGGFace2 [4] datasets. They generated synthetic degraded images by applying blur kernels (from motion and Gaussian kernels), low-light factors, and realistic noise to these clean images.

For evaluation, they created several test datasets:

  • Test-N: For noise.
  • Test-B: For blur.
  • Test-L: For low-light.
  • Test-BN: For blur and noise.
  • Test-BNL: For blur, noise, and low-light.

They also collected a real-world face image dataset with multiple degradations for qualitative evaluation.

Key Insights and Ideas

  • Identity as a Prior: Using clean reference images to extract identity information as a prior for restoration is a powerful idea. This is more robust than relying on facial landmarks or semantic maps extracted directly from degraded images.
  • NAS for Fusion: Applying NAS to design the fusion network is a smart way to optimize the combination of multi-task features, leading to better performance than hand-crafted fusion modules.
  • Specialized Operators: The introduction of FFT-op and deveiling operators specifically tailored to address blur and low-light conditions, respectively, demonstrates a nuanced approach to multi-degradation restoration.
  • Unified Multi-Task Restoration: The ability of a single network to effectively handle multiple, diverse degradations simultaneously is a significant step forward from single-task restoration models.

Key Results

NASFE demonstrated superior performance across various scenarios:

  • Single Degradation: Outperformed SOTA denoising methods by 0.6dB PSNR and 0.02 SSIM. Outperformed SOTA deblurring methods by 1.93dB PSNR and 0.04 SSIM. Outperformed SOTA low-light enhancement methods by 3dB PSNR and 0.10 SSIM.
  • Multiple Degradations (Test-BNL): Achieved a 2.1dB PSNR and 0.07 SSIM improvement over the second-best method.
  • Face Recognition: Showed a 7% improvement in Top-1 accuracy on Test-BNL compared to the second-best method, highlighting the effectiveness of the restored images for downstream tasks.
  • Qualitative Results: Visual comparisons showed NASFE producing significantly clearer and sharper face images with enhanced lighting, free from artifacts present in outputs from other methods.

Limitations and Future Work

  • Training Data Requirement: The method requires clean reference images of the same person during training to extract identity information. While this information is not needed during inference, it's a constraint for training.
  • Computational Cost: While NAS aims for efficiency, the search process itself can be computationally intensive. The paper mentions using a progressive search strategy to mitigate this.

The authors suggest that their NAS-based approach for designing the fusion network is a promising direction for future work in multi-task image restoration. The success of NASFE in handling multiple degradations also opens avenues for extending this framework to even more complex degradation scenarios.

Summary updated

Abstract

Various factors such as ambient lighting conditions, noise, motion blur, etc. affect the quality of captured face images. Poor quality face images often reduce the performance of face analysis and recognition systems. Hence, it is important to enhance the quality of face images collected in such conditions. We present a multi-task face restoration network, called Network Architecture Search for Face Enhancement (NASFE), which can enhance poor quality face images containing a single degradation (i.e. noise or blur) or multiple degradations (noise+blur+low-light). During training, NASFE uses clean face images of a person present in the degraded image to extract the identity information in terms of features for restoring the image. Furthermore, the network is guided by an identity-loss so that the identity in-formation is maintained in the restored image. Additionally, we propose a network architecture search-based fusion network in NASFE which fuses the task-specific features that are extracted using the task-specific encoders. We introduce FFT-op and deveiling operators in the fusion network to efficiently fuse the task-specific features. Comprehensive experiments on synthetic and real images demonstrate that the proposed method outperforms many recent state-of-the-art face restoration and enhancement methods in terms of quantitative and visual performance.
BibTeX
@misc{yasarla2021network,
  author = {Rajeev Yasarla and Hamid Reza Vaezi Joze and Vishal M. Patel},
  title = {{Network Architecture Search for Face Enhancement}},
  year = {2021},
  eprint = {2105.06528},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.06528},
}
Figure