Self-Correction for Human Parsing

Peike Li, Yunqiu Xu, Yunchao Wei, Yi YangIEEE TPAMI 2019 · OpenReview

TL;DR

This paper introduces Self-Correction for Human Parsing (SCHP), a novel training strategy that iteratively refines both the model and its training labels to overcome the challenges posed by noisy annotations in human parsing tasks. By employing a cyclical learning scheduler that aggregates past model weights and generates improved pseudo-labels, SCHP significantly boosts performance and achieves state-of-the-art results on challenging benchmarks.

What Problem Does This Paper Address?

Human parsing, the task of segmenting an image into fine-grained semantic parts of a human body (like arms, legs, torso, etc.), is crucial for applications like image editing and virtual reality. However, creating accurate pixel-level annotations for this task is incredibly difficult. The ambiguous boundaries between body parts and the visual similarity of certain categories often lead to noisy or incorrect labels in the ground truth data. These label noises act as a significant obstacle, hindering the performance and generalization capabilities of even the most advanced deep learning models. Existing methods typically assume clean labels, making them vulnerable to these inaccuracies. This paper tackles this critical issue by proposing a method that can learn effectively from noisy human parsing annotations.

Methods

The core of this paper is the Self-Correction for Human Parsing (SCHP) strategy. It's designed to work with an existing human parsing framework, here augmented as Augmented Context Embedding with Edge Perceiving (A-CE2P), which incorporates edge prediction to better delineate boundaries. SCHP operates in a cyclical manner:

  1. Initialization: A base human parsing model is trained on the original, potentially noisy, ground truth labels.
  2. Cyclical Learning: The training proceeds in cycles, each consisting of a fixed number of epochs (T).
  3. Online Model Aggregation: At the end of each cycle, the current model's weights are averaged with the weights of the optimal model from the previous cycle. This "self-ensembling" approach helps to smooth out the model's learning trajectory and improve generalization.
  4. Online Label Refinement: Using the aggregated model, more reliable "pseudo-masks" are generated for the training data. These pseudo-masks are essentially refined versions of the original ground truth labels, aiming to correct noise and capture more accurate boundaries. This is achieved by averaging the current pseudo-mask with the output of the current model.
  5. Iterative Improvement: The refined pseudo-masks then serve as the new ground truth for the next cycle, allowing the model to learn from progressively cleaner data. This creates a virtuous cycle where a better model generates better labels, which in turn leads to an even better model.

The A-CE2P framework itself is an extension of CE2P, introducing a consistency constraint between parsing and edge predictions to ensure geometric alignment and explicitly optimizing for mIoU alongside cross-entropy loss.

Datasets

The paper evaluates its method on two standard single-person human parsing benchmarks:

  • LIP (Large-scale Human Parsing): A large dataset with 50,462 images and 19 semantic human part labels, known for its challenging poses, occlusions, and variations.
  • PASCAL-Person-Part: A smaller dataset derived from PASCAL VOC 2010, featuring six semantic parts and significant scale variations.

Key Insights and Ideas

  • Label Noise is a Major Bottleneck: The paper strongly emphasizes that noisy annotations are a critical, often overlooked, problem in fine-grained segmentation tasks like human parsing, significantly limiting performance gains.
  • Mutual Promotion of Model and Labels: The core innovation is the idea that models and labels can iteratively improve each other. A better model can generate cleaner pseudo-labels, and cleaner labels can train a more accurate model.
  • Online, Cyclical Refinement: SCHP avoids the need for manual relabeling by performing model aggregation and label refinement in an online, cyclical fashion during training. This makes it practical and efficient.
  • Self-Ensembling for Robustness: Averaging model weights across cycles acts as a form of self-ensembling, leading to more robust models and better generalization.
  • Pseudo-Masks as "Dark Knowledge": The generated pseudo-masks are seen as containing valuable "dark knowledge" that helps correct errors in the one-hot ground truth labels.

Key Results

  • State-of-the-Art Performance: SCHP achieves new state-of-the-art results on both LIP and PASCAL-Person-Part datasets.
  • Significant mIoU Gains: On LIP, SCHP achieved an mIoU of 59.36%, outperforming the previous best by 6.26 points. On PASCAL-Person-Part, it also surpassed existing methods.
  • Improved Accuracy Across Categories: The gains are particularly notable for categories with few samples or confusing boundaries, such as scarves, sunglasses, and the distinction between left/right body parts.
  • Qualitative Improvements: Visualizations show that SCHP predictions can sometimes be even better than the ground truth labels, with more accurate boundaries and fewer errors.
  • Generalization: The method demonstrates strong generalization capabilities, achieving top results even when using standard backbones and without extensive pre-training.

Limitations and Future Work

  • Initialization Sensitivity: The paper notes that the initial performance of the model is important; if the initial training results are too inaccurate, they might negatively impact the iterative process. They mitigate this by starting SCHP after the initial training loss has started to flatten.
  • Generalization to Other Tasks: While the authors believe SCHP is a general strategy, its effectiveness on tasks other than segmentation (like regression-based detection) is not explicitly demonstrated.
  • Future Work: The authors suggest extending SCHP to multi-person human parsing and video human parsing tasks. They also propose exploring its application in semi-supervised learning settings where test images could be used for refinement.

Summary updated

Abstract

Labeling pixel-level masks for fine-grained semantic segmentation tasks, e.g. human parsing, remains a challenging task. The ambiguous boundary between different semantic parts and those categories with similar appearance usually are confusing, leading to unexpected noises in ground truth masks. To tackle the problem of learning with label noises, this work introduces a purification strategy, called Self-Correction for Human Parsing (SCHP), to progressively promote the reliability of the supervised labels as well as the learned models. In particular, starting from a model trained with inaccurate annotations as initialization, we design a cyclically learning scheduler to infer more reliable pseudo-masks by iteratively aggregating the current learned model with the former optimal one in an online manner. Besides, those correspondingly corrected labels can in turn to further boost the model performance. In this way, the models and the labels will reciprocally become more robust and accurate during the self-correction learning cycles. Benefiting from the superiority of SCHP, we achieve the best performance on two popular single-person human parsing benchmarks, including LIP and Pascal-Person-Part datasets. Our overall system ranks 1st in CVPR2019 LIP Challenge. Code is available at https://github.com/PeikeLi/Self-Correction-Human-Parsing.
BibTeX
@article{li2019self,
  author = {Peike Li and Yunqiu Xu and Yunchao Wei and Yi Yang},
  title = {{Self-Correction for Human Parsing}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2019},
  url = {https://openreview.net/forum?id=6xpquysiri},
}
Figure