Face Parsing From RGB and Depth Using Cross-Domain Mutual Learning

Jihyun Lee, Binod Bhattarai, Tae-Kyun KimWorkshops @ CVPR 2021 · OpenReview

Abstract

Existing methods of face parsing have proven effective at classifying each pixel of an RGB image into different facial components. However, there is a lack of face parsing research that utilizes depth domain. To the best of our knowledge, we present the first study to exploit 2.5D data for face parsing. We introduce a novel framework to jointly learn (1) RGB face parsing, (2) depth face parsing and (3) RGB-to-depth domain translation, which can be effective even when only a small amount of annotated depth data is available for training. To this end, we also create the first RGB-D face parsing benchmarks based on CelebAMask-HQ, LaPa and Helen by utilizing an off-the-shelf 3D head reconstruction model. Overall, our approach makes two main contributions. First, our method leverages mutual learning between RGB and depth face parsing, which enables bidirectional knowledge distillation between the two data domains. Second, our method utilizes end-to-end learning of RGB-to-depth domain translation and depth face parsing, which can help overcome the scarcity of annotated depth data. We perform extensive experiments to validate the effectiveness of our method, in which we achieve state-of-the-art results in RGB face parsing.

Introduction

Face parsing—the task of labeling every pixel in a face image as a specific component like "nose," "eye," or "lip"—is critical for applications like digital makeup, animation, and face synthesis. However, traditional RGB-based methods face two major hurdles: environmental sensitivity and data scarcity. RGB images struggle in extreme lighting conditions and do not explicitly capture the 3D geometry of the face. Furthermore, while depth maps are less sensitive to lighting, there were previously no large-scale, annotated RGB-D (Red, Green, Blue + Depth) datasets specifically for face parsing.

Proposed Framework

The researchers developed a unified framework consisting of three sub-networks trained in an end-to-end fashion: an RGB Face Parsing Network, a Depth Face Parsing Network, and an RGB-to-Depth Translation Network based on the Pix2Pix architecture.

The proposed framework architecture featuring the RGB-to-depth translation network and the mutual learning distillation process between RGB and depth branches.

Cross-Domain Mutual Learning

The core innovation is Cross-Domain Mutual Learning. Unlike typical "teacher-student" models where knowledge flows one way, this method uses a bidirectional distillation loss ($L_{ML}$). This encourages the RGB and Depth networks to mimic each other's activations in their final layers. Additionally, an end-to-end ($L_{E2E}$) loss allows the depth parser to train on synthetic depth maps generated by the translation network. This is crucial because it allows the model to benefit from the vast amount of available annotated RGB data, even when real depth annotations are scarce.

Datasets

Since no RGB-D face parsing benchmarks existed, the authors created three new ones by applying an off-the-shelf 3D head reconstruction model to existing popular datasets:

  • CelebAMask-HQ-D: ~30,000 images.
  • LaPa-D: ~21,000 images.
  • Helen-D: ~2,300 images.

These datasets provide "pseudo-ground truth" depth maps paired with the original RGB images and semantic labels, allowing for the first comprehensive study of 2.5D face parsing.

Key Results

The proposed method achieved state-of-the-art (SOTA) results in RGB face parsing and provided the first-ever benchmarks for depth-based parsing. On the CelebAMask-HQ dataset, the method improved the mean F1-score of the BiSeNet baseline by 8.1% and the EAGRNet baseline by 1.2%.

Qualitative results comparing the BiSeNet baseline against the proposed method. The red boxes highlight areas where the mutual learning approach significantly improves segmentation accuracy.

Beyond segmentation, the joint learning approach improved the translation network itself, reducing the L1 distance error in predicted depth maps by approximately 20% compared to a standalone Pix2Pix model. The depth-only parsing results were also found to be surprisingly close to RGB results, suggesting that depth maps are a highly viable alternative for face parsing in low-light indoor environments.

Conclusion and Limitations

By treating depth as privileged information, the network learns to consider underlying 3D geometry even when only an RGB image is available at test time. This bidirectional approach captures a more holistic representation of the face.

A primary limitation is the reliance on pseudo-ground truth generated by 3D reconstruction models rather than physical sensors (like LiDAR). Future research will focus on acquiring real-world RGB-D datasets with manual annotations to further validate these cross-domain benefits and investigate how to close the gap between reconstructed and captured depth.

Summary updated

Abstract

Existing methods of face parsing have proven effective at classifying each pixel of an RGB image into different facial components. However, there is a lack of face parsing research that utilizes depth domain. To the best of our knowledge, we present the first study to exploit 2.5D data for face parsing. We introduce a novel framework to jointly learn (1) RGB face parsing, (2) depth face parsing and (3) RGB-to-depth domain translation, which can be effective even when only a small amount of annotated depth data is available for training. To this end, we also create the first RGB-D face parsing benchmarks based on CelebAMask-HQ, LaPa and Helen by utilizing an off-the-shelf 3D head reconstruction model. Overall, our approach makes two main contributions. First, our method leverages mutual learning between RGB and depth face parsing, which enables bidirectional knowledge distillation between the two data domains. Second, our method utilizes end-to-end learning of RGB-to-depth domain translation and depth face parsing, which can help overcome the scarcity of annotated depth data. We perform extensive experiments to validate the effectiveness of our method, in which we achieve state-of-the-art results in RGB face parsing. As far as we know, we also report the first results on face parsing from depth data. All experiments are conducted on our new RGB-D face parsing datasets, which are publicly available at https://github.com/jyunlee/CelebAMask-HQ-D_LaPa-D_Helen-D.
BibTeX
@inproceedings{lee2021face,
  author = {Jihyun Lee and Binod Bhattarai and Tae-Kyun Kim},
  title = {{Face Parsing From RGB and Depth Using Cross-Domain Mutual Learning}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=Ej0DzuEblN9},
}
Figure