Deep Representation Calibrated Bayesian Neural Network for Semantically Explainable Face Inpainting and Editing

Hao Xiong, Chaoyue Wang, Xinchao Wang, Dacheng TaoIEEE Access 2020 · OpenReview

Abstract

Image inpainting seeks to fill in corrupted areas with pixels that have a similar texture and content with its surroundings. For high-structured data, e.g., human face, some recent works can achieve quite realistic results. However, almost all existing methods learned a determined mapping from a corrupted input to the final result, yet ignored the potential multiple plausible solutions of the same input. Furthermore, they have not explored the underlying connections between those plausible solutions and semantic conditions. In this work, we propose a novel deep representation calibrated Bayesian neural network (DRCBNN) for semantically explainable face inpainting and editing. By leveraging the advantages that Bayesian decision theory deals with uncertainty, the proposed framework exploits deep representation into Bayesian decision theory and derive a deep representation calibrated evidence lower bound (ELBO). In comparison with traditional ELBO in BNN, the newly calibrated ELBO is a more task-specific loss function. After optimizing the newly calibrated ELBO, it allows to inference desired inpainting outputs in accordance with specific semantics. Finally, experiments demonstrated that our method can produce multiple semantics-aware inpainting outputs and outperforms the state-of-the-arts.

Introduction

Image inpainting—the process of reconstructing missing or corrupted parts of an image—has evolved significantly with the advent of deep learning. While modern models can generate realistic textures, they often struggle with high-structured data like human faces. Most existing frameworks are deterministic, producing a single "best-guess" output for any given corruption. This approach fails to account for the inherent ambiguity of missing data; a missing patch of a face could realistically contain various eye shapes, skin textures, or facial hair configurations.

Furthermore, standard inpainting models act as "black boxes" regarding semantic content. Users have little control over the specific features generated in the restored areas. The Deep Representation Calibrated Bayesian Neural Network (DRCBNN) addresses these flaws by fusing Bayesian decision theory with deep latent representations, enabling semantically explainable and controllable inpainting.

A corrupted input image where the eyes and mouth are obscured by white masks, representing the challenge of structured inpainting.

Methodology: The DRCBNN Framework

The core innovation of this research is the transition from fixed-weight networks to a Bayesian framework. In a Bayesian Neural Network (BNN), weights are treated as probability distributions rather than static values, allowing the model to quantify uncertainty.

Calibrated Evidence Lower Bound (ELBO)

The authors derive a task-specific loss function: the Deep Representation Calibrated ELBO. Unlike standard BNN ELBOs that focus primarily on weight uncertainty, this calibrated version incorporates a term for deep latent representations ($z$). This ensures that the model captures high-level semantic features rather than just low-level pixel textures.

Architecture and Inference

The DRCBNN utilizes a UNet-like structure for the primary inpainting task, coupled with two auxiliary networks:

  1. $U_\phi$: A convolutional network that estimates the parameters of the latent distribution.
  2. $C_\omega$: A semantic recognition network that ensures the generated output aligns with desired attributes.

During inference, the model can perform "blind" inpainting by sampling from the latent space or "semantics-aware" inpainting. In the latter, the latent variable is optimized to match user-provided labels, such as "mustache" or "makeup," allowing for precise editing of the restored regions.

An example of semantics-aware inpainting where the model has filled in missing regions while applying specific attributes like makeup and facial hair.

Experimental Evaluation

The researchers evaluated DRCBNN against state-of-the-art baselines, including PConv and Pluralistic Image Completion (PIC), using the CelebA, UTKFace, and Celeba-HQ datasets.

Quantitative Performance

On the Celeba-HQ dataset, DRCBNN demonstrated superior structural and signal quality:

  • SSIM: 0.8745
  • PSNR: 36.62

Beyond standard metrics, the authors employed a Semantic Dice Coefficient to measure the accuracy of attribute generation. DRCBNN achieved a score of 76.00, significantly outperforming competitors and proving its efficacy in generating requested facial features.

Diversity and Realism

Using the LPIPS metric to measure perceptual diversity, the study showed that DRCBNN produces a wider and more realistic variety of solutions than previous pluralistic models. This is particularly evident in complex areas like hair and facial accessories, where the model avoids "blurry averages" in favor of distinct, plausible features.

Conclusion

The DRCBNN framework represents a significant step forward in making image inpainting both probabilistic and explainable. By calibrating the Bayesian ELBO with deep representations, the model moves beyond simple pixel restoration to true semantic editing. While challenges remain in scaling to ultra-high resolutions and improving optimization speed during inference, the ability to provide controlled, multiple "explanations" for corrupted data sets a new standard for the field.

Summary updated

Abstract

Image inpainting seeks to fill in corrupted areas with pixels that have a similar texture and content with its surroundings. For high-structured data, e.g., human face, some recent works can achieve quite realistic results. However, almost all existing methods learned a determined mapping from a corrupted input to the final result, yet ignored the potential multiple plausible solutions of the same input. Furthermore, they have not explored the underlying connections between those plausible solutions and semantic conditions. In this work, we propose a novel deep representation calibrated Bayesian neural network (DRCBNN) for semantically explainable face inpainting and editing. By leveraging the advantages that Bayesian decision theory deals with uncertainty, the proposed framework exploits deep representation into Bayesian decision theory and derive a deep representation calibrated evidence lower bound (ELBO). In comparison with traditional ELBO in BNN, the newly calibrated ELBO is a more task-specific loss function. After optimizing the newly calibrated ELBO, it allows to inference desired inpainting outputs in accordance with specific semantics. Finally, experiments demonstrated that our method can produce multiple semantics-aware inpainting outputs and outperforms the state-of-the-arts.
BibTeX
@inproceedings{xiong2020deep,
  author = {Hao Xiong and Chaoyue Wang and Xinchao Wang and Dacheng Tao},
  title = {{Deep Representation Calibrated Bayesian Neural Network for Semantically Explainable Face Inpainting and Editing}},
  booktitle = {IEEE Access},
  year = {2020},
  url = {https://openreview.net/forum?id=AUUGT9gmvvQ},
}
Figure