Abstract
Despite the noticeable progress in perceptual tasks like detection, instance segmentation and human parsing, computers still perform unsatisfactorily on visually understanding humans in crowded scenes, such as group behavior analysis, person re-identification and autonomous driving. To this end, models need to comprehensively perceive the semantic information and the differences between instances in a multi-human image, which is recently defined as the multi-human parsing task. In this paper, we present a new large-scale database "Multi-Human Parsing (MHP)" for algorithm development and evaluation, and advances the state-of-the-art in understanding humans in crowded scenes. MHP contains 25,403 elaborately annotated images with 58 fine-grained semantic category labels, involving 2-26 persons per image and captured in real-world scenes from various viewpoints, poses, occlusion, interactions and background. We further propose a novel deep Nested Adversarial Network (NAN) model for multi-human parsing. NAN consists of three Generative Adversarial Network (GAN)-like sub-nets, respectively performing semantic saliency prediction, instance-agnostic parsing and instance-aware clustering. These sub-nets form a nested structure and are carefully designed to learn jointly in an end-to-end way. NAN consistently outperforms existing state-of-the-art solutions on our MHP and several other datasets, and serves as a strong baseline to drive the future research for multi-human parsing.
What Problem Does This Paper Address?
While computer vision has made huge strides in detecting people (bounding boxes) and instance segmentation (identifying person silhouettes), multi-human parsing remains a significant hurdle. This task requires a model to not only identify every person in a crowd but also label their specific body parts (arms, legs, face) and fashion items (hats, coats, shoes) at the pixel level.
The challenge is twofold:
- Complexity: Real-world scenes are crowded. People overlap, interact, and appear in diverse poses or under heavy occlusion.
- Data Scarcity: Existing datasets were either too small, lacked fine-grained labels, or focused only on single-person images.
The authors argue that mastering this task is essential for high-level applications like autonomous driving, group behavior analysis, and virtual reality.
Methods: The Nested Adversarial Network (NAN)
The core technical contribution is the Nested Adversarial Network (NAN). Unlike traditional methods that rely on slow, multi-stage region proposals, NAN uses a "nested" structure of three Generative Adversarial Network (GAN)-like sub-nets that boost each other's performance:
- Semantic Saliency Prediction: The first sub-net acts as a "prior," locating the most noticeable human regions and separating the foreground from the background.
- Instance-Agnostic Parsing: The second sub-net takes the original image plus the saliency map to perform fine-grained part labeling. At this stage, it identifies parts (e.g., an "arm") without yet assigning them to a specific person.
- Instance-Aware Clustering: The final sub-net assigns those parsed parts to specific individuals by predicting instance locations and the total number of people.
By using adversarial losses (discriminators) at each stage, the model ensures that the generated parsing maps look "real" and remain consistent with the original image. The entire stack is trained end-to-end, allowing for a single forward pass during inference.
Datasets: MHP v2.0
The authors introduce MHP v2.0, which significantly expands previous efforts:
- Scale: 25,403 images containing 2–26 persons per image.
- Granularity: 58 fine-grained semantic category labels (e.g., distinguishing between a "t-shirt," "polo shirt," and "singlet").
- Diversity: Captured in real-world settings with varying viewpoints, occlusions, and backgrounds.
The model was also evaluated on several legacy benchmarks to prove its versatility, including MHP v1.0, LIP, PASCAL-Person-Part, and Buffy.
Key Results and Insights
- Task Decomposition: By breaking the task into three simpler sub-tasks (Saliency → Agnostic Parsing → Clustering), the model becomes more accurate and easier to train.
- State-of-the-Art Performance: On the MHP v2.0 validation set, NAN achieved an Average Precision ($AP^p$) of 24.83%, significantly outperforming the previous baseline, MH-Parser (18.05%), and Mask R-CNN (14.50%).
- Robustness in Crowds: In "Inter10%" scenarios (images with the highest interaction/crowding), NAN maintained a lead, proving it handles overlapping people better than traditional instance segmentation.
- Efficiency: By avoiding the "detect-then-crop" pipeline, NAN processes the whole scene holistically and faster, clocking in at roughly 1 second per image.
Limitations and Future Work
Despite the progress, the authors note several remaining challenges. Extreme occlusion remains difficult, as does the accurate parsing of very small categories like sunglasses or watches. Future research may incorporate temporal data from video to resolve ambiguities in crowded scenes and focus on even more complex human-human and human-object interactions.
Summary updated