Keypoints as Dynamic Centroids for Unified Human Pose and Segmentation

Niaz Ahmad, Jawad Khan, Kang G. Shin, Youngmoon Lee, Guanghui WangIJCAI 2025 · DOI

TL;DR

This paper introduces Keypoints as Dynamic Centroids (KDC), a novel unified framework for simultaneously estimating human pose and segmenting individual instances. KDC leverages keypoints as dynamic centroids to efficiently cluster pixels for segmentation, outperforming existing methods in challenging scenarios like crowded scenes and rapid movements.

What Problem Does This Paper Address?

Accurately identifying human poses and segmenting individual bodies in images and videos is crucial for applications like human-computer interaction and visual analysis. However, existing methods often struggle with:

  • Overlapping and occluded individuals: Differentiating between multiple people when they are close together or partially hidden is difficult.
  • Rapidly changing poses: Dynamic movements can make it hard for algorithms to maintain accurate tracking and segmentation.
  • Computational complexity: Many state-of-the-art approaches are too slow for real-time applications, especially when dealing with many people.

Current approaches often tackle pose estimation and segmentation separately or rely on computationally expensive person detectors. This paper aims to bridge this gap by proposing a unified, efficient, and scalable method.

Methods

The KDC framework comprises two main networks: PoseNet and SegNet, integrated by a novel "PoseSeg" module.

  • PoseNet with Keypoint Disks and KeyCentroids:
    • Keypoint Disks (KHDR): Instead of just heatmaps, PoseNet generates keypoint heatmaps using a "disk representation." This helps define a region of interest for each keypoint, improving localization accuracy, especially for occluded or distant keypoints.
  • KeyCentroid: This is a novel refinement step. For each keypoint, a vector field is learned within its disk, pointing towards the keypoint's true spatial coordinates. This "KeyCentroid" vector helps to precisely localize keypoints and improve their confidence scores.
  • SegNet with MaskCentroids:
    • MaskCentroid: This is the core innovation for segmentation. High-confidence keypoints identified by PoseNet are used as dynamic centroids. These centroids are then used to cluster pixels belonging to specific human instances in an embedding space. This dynamic nature allows the system to adapt to rapid changes in human body movements and occlusions.
  • Instance-wise Gaussian Optimization: A Gaussian smoothing technique is applied at the instance level to refine semantic maps and improve segmentation accuracy, particularly in crowded scenes.
  • PoseSeg Module: This module integrates the outputs from PoseNet and SegNet, creating a unified representation for both tasks. It uses a priority queue to manage keypoints and their coordinates, facilitating instance detection and pose estimation.

Datasets

The authors evaluated KDC on three standard benchmarks:

  • COCO (Common Objects in Context): A widely used dataset for object detection, segmentation, and keypoint detection.
  • CrowdPose: Specifically designed to challenge methods in crowded scenarios with many overlapping individuals.
  • OCHuman: Another dataset focusing on human pose estimation in challenging, often occluded, environments.

The model was trained end-to-end on the COCO keypoint and segmentation training sets.

Key Insights and Ideas

  • Keypoints as dynamic anchors for segmentation: The most significant insight is using keypoints not just for pose but as dynamic, learned anchors to guide pixel-wise segmentation. This elegantly links the two tasks.
  • KeyCentroid for precise localization: The idea of learning a vector field within a keypoint disk to refine its exact location is a clever way to boost accuracy beyond simple heatmap peak detection.
  • Dynamic MaskCentroids adapt to motion: Using learned keypoints as dynamic centroids for segmentation allows the system to be more robust to the constant changes and occlusions inherent in real-world human movement.
  • Unified approach simplifies pipelines: By combining pose and segmentation into a single framework, KDC reduces the need for separate models and complex post-processing, leading to greater efficiency.

Key Results

KDC demonstrates strong performance across all three datasets:

  • Keypoint Estimation: KDC outperforms recent single-stage and top-down methods on COCO, achieving an mAP of 76.1% with ResNet-152. It also shows competitive results on CrowdPose and OCHuman.
  • Instance Segmentation: KDC achieves a top accuracy of 47.6 mAP on COCO, significantly improving upon Mask R-CNN. It also shows substantial gains on OCHuman.
  • Efficiency: The paper highlights KDC's lower parameter count, higher FPS, and reduced computational complexity compared to models like Mask R-CNN and PersonLab, making it suitable for real-time applications.

Limitations and Future Work

  • Limitations: While KDC shows impressive results, the paper notes that its segmentation performance might still be challenged in extremely dense and highly entangled scenarios. The reliance on learned centroids means that if keypoints are severely mispredicted, segmentation could be affected.
  • Future Work: The authors suggest that further exploration of dynamic centroid learning and potentially incorporating temporal information could lead to even more robust performance in video sequences. Investigating the impact of different backbone architectures and further optimizing the PoseSeg module for even greater efficiency are also potential avenues.

Summary updated

BibTeX
@inproceedings{ahmad2025keypoints,
  author = {Niaz Ahmad and Jawad Khan and Kang G. Shin and Youngmoon Lee and Guanghui Wang},
  title = {{Keypoints as Dynamic Centroids for Unified Human Pose and Segmentation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/75},
}
Figure