A New Dataset and Boundary-Attention Semantic Segmentation for Face Parsing

Yinglu Liu, Hailin Shi, Hao Shen, Yue Si, Xiaobo Wang, Tao MeiAAAI 2020 · DOI

Abstract

Face parsing, the task of assigning semantic labels to facial pixels, is foundational for digital makeup and face synthesis. However, progress is hindered by the lack of large-scale, diverse datasets and the inherent difficulty of accurately segmenting facial boundaries. This paper introduces LaPa, a massive new dataset of over 22,000 images with dense 106-point landmarks and 11-category masks. Alongside the dataset, the authors propose Boundary-Attention Semantic Segmentation (BASS), a multi-branch deep learning architecture designed to mitigate "boundary confusion" through a dedicated boundary-aware branch and a weighted loss function. Experimental results demonstrate that BASS achieves state-of-the-art performance on the Helen benchmark and provides a strong baseline for the new LaPa dataset.

Introduction

Face parsing involves assigning a semantic label (such as "nose," "upper lip," or "hair") to every pixel in a facial image. Despite its importance in computer vision, two major hurdles persist: data scarcity and boundary confusion. High-quality pixel labeling is prohibitively expensive, leading to datasets that are either small or lack diversity in pose and expression. Furthermore, standard segmentation models often fail at the transition zones between facial features, resulting in "blurry" or inaccurate boundaries. This paper addresses these issues by releasing the LaPa dataset and introducing the BASS model architecture.

The LaPa Dataset

The LaPa (Landmark-guided Face Parsing) dataset consists of 22,176 images, each annotated with 11-category pixel-level labels and 106-point dense landmarks. Unlike previous datasets, LaPa focuses on "in-the-wild" variety, capturing extreme poses, expressions, and occlusions that challenge existing models.

Examples from the LaPa dataset showing diverse poses, landmarks, and semantic segmentation masks.

Annotation Framework

To generate such a large dataset efficiently, the authors developed a high-efficiency pipeline to avoid the slow pace of manual pixel-level labeling:

  • Dense Landmark Annotation (DLA): A semi-automatic tool where an auxiliary model suggests 106 landmark positions, and humans only "tweak" the difficult cases.
  • Pixel-Level Parsing Annotation (PPA): This module takes the landmarks and automatically generates masks using category-wise fitting (e.g., using parabolas for lips) and coarse-to-fine segmentation for hair and skin.
The semi-automatic annotation pipeline, combining landmark detection with automated mask fitting.

The BASS Model

The Boundary-Attention Semantic Segmentation (BASS) network employs three parallel branches to improve edge precision:

  1. Semantic Branch: A standard ResNet-101 backbone with a Pyramid Spatial Pooling (PSP) module to capture global context.
  2. Boundary-Aware Branch: Specifically trained to perform a two-category (boundary vs. non-boundary) segmentation task.
  3. Boundary-Attention Semantic Branch: This branch fuses features from the first two branches. Crucially, it uses a Boundary-Attention Semantic Loss. This loss function uses a parameter ($\alpha$) to heavily penalize errors on boundary pixels, forcing the model to "pay more attention" to the edges.

In their ablation studies, the authors found that simply concatenating boundary features with semantic features worked better than adding or multiplying them, suggesting that the network benefits from having both "what" and "where" information available in distinct channels.

Key Results and Insights

  • State-of-the-Art on Helen: BASS achieved an overall F1-score of 93.1%, surpassing previous leading methods.
  • LaPa Performance: On the new LaPa dataset, the full BASS model achieved a mean F1-score of 89.79%.
  • Ablation Success: Adding the boundary-aware branch improved the baseline, but the boundary-attention loss (the $\alpha$ parameter) provided a much larger jump, particularly for features with sharp edges like eyes and eyebrows.
  • Qualitative Improvement: Visual results show that BASS handles occlusions (like hair covering an eye) much more cleanly than models that lack boundary attention.

Limitations and Future Work

The authors note that the nose remains the most challenging part to segment. Unlike the eyes or mouth, the nose often lacks a high-contrast boundary against the rest of the face skin, making the boundary-attention mechanism less effective there. Future work may involve making the DLA module entirely autonomous as auxiliary models improve, further reducing the need for manual intervention.

Summary updated

BibTeX
@inproceedings{liu2020new,
  author = {Yinglu Liu and Hailin Shi and Hao Shen and Yue Si and Xiaobo Wang and Tao Mei},
  title = {{A New Dataset and Boundary-Attention Semantic Segmentation for Face Parsing}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6832},
}
Figure