Joint Human Pose Estimation and Instance Segmentation with PosePlusSeg

Niaz Ahmad, Jawad Khan, Jeremy Yuhyun Kim, Youngmoon LeeAAAI 2022 · DOI

Abstract

PosePlusSeg is a novel bottom-up framework designed to jointly tackle human pose estimation and instance segmentation without the need for traditional bounding box detectors. By integrating a "Strong Keypoint Heat Map" (SKHM) and a "Body Heat Map" (BHM), the model achieves state-of-the-art performance on the COCO dataset, reaching 0.728 mAP for pose and 0.445 mAP for segmentation. Crucially, the architecture maintains the efficiency required for real-time applications, processing frames at approximately 30 FPS.

Introduction

Understanding human subjects in computer vision typically involves two distinct tasks: Pose Estimation (identifying skeletal joints) and Instance Segmentation (pixel-level masking). Most high-performing models employ a "top-down" approach, first detecting individuals via bounding boxes and then analyzing each box. While accurate, this method is computationally expensive as its complexity scales with the number of people in a scene.

In contrast, "bottom-up" methods detect all joints in an image simultaneously and then group them. While faster, these methods have historically struggled with accuracy in crowded scenes or complex body configurations. PosePlusSeg addresses these limitations by proposing a bottom-up model where keypoint data directly informs the segmentation process, eliminating the "detect-then-analyze" bottleneck.

Methodology

PosePlusSeg utilizes a CNN backbone (e.g., ResNet-101 or ResNet-152) and splits into two parallel pipelines for pose and segmentation.

Pose Estimation Pipeline

The pose pipeline introduces two critical components:

  • Strong Keypoint Heat Map (SKHM): Instead of predicting single-pixel points, the model predicts disks (radius $R=32$ pixels) around keypoints. This provides a denser training signal, improving localization.
  • Body Heat Map (BHM): This map identifies general human body regions and is used to "boost" keypoint confidence scores, effectively filtering out background false positives.

Instance Segmentation Pipeline

The segmentation pipeline relies on:

  • Mask Offset (MO): Each pixel predicts its relative distance to a "center of attraction" (typically the right shoulder), facilitating efficient pixel grouping.
  • Segmentation Mask (SM): This predicts the overall 2D shape structure of the human body.

Integration and Refinement

A lightweight 1x1 convolutional "Refine Network" polishes the raw heatmaps. Finally, the Pose and Instance Segmentation (PIS) algorithm merges these outputs, using refined keypoints as anchors for the segmentation masks to ensure perfect alignment between the skeleton and the body mask.

Visual pipeline of PosePlusSeg showing intermediate heatmaps and final joint pose and segmentation output: (a) Input, (b) SKHM, (c) BHM, (d) Pose Estimation, (e) Mask Offset, (f) Segmentation Mask, (g) Instance Segmentation Mask, (h) Final PIS result.

Experimental Results

The model was evaluated on the COCO (Common Objects in Context) dataset, specifically the human category, using standard minival and test-dev splits.

Performance Metrics

PosePlusSeg demonstrates significant improvements over both top-down and bottom-up competitors:

  • Pose Estimation: Achieved 0.728 mAP on the COCO test set, outperforming top-down methods like CPN (0.721) and bottom-up methods like PersonLab (0.687).
  • Instance Segmentation: Achieved 0.445 mAP on the COCO test set, a substantial gain over Mask R-CNN (0.371) and Pose2Seg (0.555 on minival, though PosePlusSeg demonstrates superior scaling).

Efficiency

On an NVIDIA RTX GPU, PosePlusSeg achieves inference speeds between 28ms and 34ms (~30 FPS). Because it is a bottom-up approach, its runtime remains relatively constant regardless of the number of people in the image, making it highly scalable for crowded environments.

Discussion and Key Insights

A central innovation of PosePlusSeg is the use of keypoints as anchors for masks. By using the right shoulder as a centroid for pixel clustering instead of a bounding box, the model creates a tighter coupling between the skeleton and body shape. Furthermore, the Body Heat Map acts as a spatial filter, significantly reducing errors caused by "entangled limbs" in crowded scenes.

Limitations and Future Work

Despite its strengths, extremely crowded scenarios still pose challenges for correct body part association. Future research may explore alternative "centers of attraction" for more robust clustering or adapt the framework for 3D pose estimation and volumetric segmentation. Additionally, further optimization of the Refine Network could enable deployment on resource-constrained edge devices.

Conclusion

PosePlusSeg represents a significant step forward for bottom-up human analysis. By jointly optimizing for pose and segmentation and leveraging novel heatmap structures, it achieves state-of-the-art accuracy without sacrificing the speed necessary for real-time deployment.

Summary updated

BibTeX
@inproceedings{ahmad2022joint,
  author = {Niaz Ahmad and Jawad Khan and Jeremy Yuhyun Kim and Youngmoon Lee},
  title = {{Joint Human Pose Estimation and Instance Segmentation with PosePlusSeg}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i1.19880},
}
Figure