Nondiscriminatory Treatment: a straightforward framework for multi-human parsing

Min Yan, Guoshan Zhang, Tong Zhang, Yueming Zhang

Abstract

Multi-human parsing aims to segment every body part of every human instance. Nearly all state-of-the-art methods follow the "detection first" or "segmentation first" pipelines. Different from them, we present an end-to-end and box-free pipeline from a new and more human-intuitive perspective. In training time, we directly do instance segmentation on humans and parts. More specifically, we introduce a notion of "indiscriminate objects with categories" which treats humans and parts without distinction and regards them both as instances with categories. In the mask prediction, each binary mask is obtained by a combination of prototypes shared among all human and part categories. In inference time, we design a brand-new grouping post-processing method that relates each part instance with one single human instance and groups them together to obtain the final human-level parsing result. We name our method as Nondiscriminatory Treatment between Humans and Parts for Human Parsing (NTHP). Experiments show that our network performs superiorly against state-of-the-art methods by a large margin on the MHP v2.0 and PASCAL-Person-Part datasets.

Introduction

Multi-human parsing—the task of segmenting every body part of every person in an image—is notoriously difficult due to frequent occlusions, irregular body poses, and complex interactions between people. Traditional approaches generally fall into two camps: "Detection First" (top-down) and "Segmentation First" (bottom-up).

Top-down models detect a person with a bounding box and then parse parts within that box. However, bounding boxes are axis-aligned rectangles that often include "excessive" features from the background or other people, making the final result heavily dependent on the quality of the initial box. Bottom-up models perform pixel-level classification and then group pixels into instances, which often fails at boundaries where people overlap.

Comparison of traditional detection-based approach (b) vs. the proposed mask-based indiscriminate treatment (c, d)

The authors argue these methods aren't "human-intuitive." When we look at a crowd, we see a collection of parts that belong to a whole, rather than a series of boxes or a sea of independent pixels.

Methodology: Nondiscriminatory Treatment (NTHP)

The proposed NTHP framework reframes multi-human parsing as a unified instance segmentation task.

Indiscriminate Objects with Categories

The core philosophy is to treat a "human" instance and a "left arm" instance without distinction during the initial stages of the network. Both are simply "objects" to be segmented. This "nondiscriminatory treatment" allows the model to learn shared features across different scales.

Unified Mask Prediction Based on Prototypes (UMPP)

NTHP uses a Feature Pyramid Network (FPN) to handle different scales. The network learns a set of "basis masks" (prototypes) shared across all categories (humans and parts). For every grid cell, the model predicts a set of coefficients. The final binary mask for an instance is generated by a linear combination of these prototypes weighted by the predicted coefficients. This allows humans and parts to share a "visual vocabulary," where part features help refine human masks and vice versa.

Handling Translation-Invariance and Grouping

Standard convolutions are translation-invariant. To address this, NTHP uses CoordConv, adding coordinate information to feature maps to distinguish between identical-looking parts in different image locations. Since humans and parts are predicted independently, a post-processing step uses an overlap ratio to "reattach" parts: a part is assigned to a human instance if the majority of its area falls within that human's predicted mask.

Experiments and Results

The model was evaluated on the MHP v2.0 and PASCAL-Person-Part datasets.

Key Insights

  • Mutual Benefit: Sharing prototypes between humans and parts leads to more robust features. For example, a "torso" prototype helps define the boundaries of the "human" instance.
  • Scale-Aware Assignment: Human instances (large) are assigned to the topmost level of the FPN, while part instances (small) are assigned to lower levels, naturally handling massive scale variance.
  • Box-Free Advantage: By avoiding RoI-Align and bounding boxes, the model better handles irregular poses where limbs extend far from the person's center.

Performance

NTHP significantly outperformed previous state-of-the-art (SOTA) methods:

  • MHP v2.0: Achieved 51.1 $AP^p_{50}$, a 5.8-point margin over the previous best method (RP R-CNN).
  • PASCAL-Person-Part: Achieved 47.1 $AP^r_{vol}$, outperforming the "Unified" framework by 4 points.

Conclusion

NTHP introduces a novel one-stage, box-free framework that treats humans and their body parts as equal "instances with categories." By utilizing shared mask prototypes and a straightforward overlap-based grouping strategy, the model achieves state-of-the-art results while avoiding the architectural complexity of traditional two-stage pipelines. Future work may explore making the grouping step end-to-end and utilizing larger backbones like Swin Transformers.

Summary updated

Abstract

Multi-human parsing aims to segment every body part of every human instance. Nearly all state-of-the-art methods follow the "detection first" or "segmentation first" pipelines. Different from them, we present an end-to-end and box-free pipeline from a new and more human-intuitive perspective. In training time, we directly do instance segmentation on humans and parts. More specifically, we introduce a notion of "indiscriminate objects with categorie" which treats humans and parts without distinction and regards them both as instances with categories. In the mask prediction, each binary mask is obtained by a combination of prototypes shared among all human and part categories. In inference time, we design a brand-new grouping post-processing method that relates each part instance with one single human instance and groups them together to obtain the final human-level parsing result. We name our method as Nondiscriminatory Treatment between Humans and Parts for Human Parsing (NTHP). Experiments show that our network performs superiorly against state-of-the-art methods by a large margin on the MHP v2.0 and PASCAL-Person-Part datasets.
BibTeX
@misc{yan2021nondiscriminatory,
  author = {Min Yan and Guoshan Zhang and Tong Zhang and Yueming Zhang},
  title = {{Nondiscriminatory Treatment: a straightforward framework for multi-human parsing}},
  year = {2021},
  eprint = {2101.10913},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2101.10913},
}
Figure