Adaptive Hypergraph Neural Network for Multi-Person Pose Estimation

Xixia Xu, Qi Zou, Xue LinAAAI 2022 · DOI

Abstract

Multi-person pose estimation (MPPE) often relies on fixed skeletal structures that fail to capture complex, high-order joint dependencies. We present the Adaptive Hypergraph Neural Network (AD-HNN), a two-stage framework that dynamically learns body joint relationships. By employing an adaptive incidence matrix, the model adjusts its internal representation of the human body based on specific poses and occlusions, achieving state-of-the-art results on MS-COCO, MPII, and CrowdPose datasets.

Introduction

Locating keypoints such as elbows, knees, and wrists for multiple individuals in a single image is a fundamental challenge in computer vision. While deep learning has significantly advanced MPPE, traditional models are often limited by two factors: the use of simple pairwise joint relationships and the reliance on static "kinematic priors." These fixed skeletal maps do not account for the dynamic nature of human movement or the complexities of self-occlusion and crowded environments. This research introduces a method to "re-wire" the model's understanding of human topology on the fly using adaptive hypergraphs.

Methodology

The AD-HNN framework operates in two distinct stages to refine pose estimation through both visual semantics and structural learning.

Stage 1: Initial Localization and Semantic Interaction

The process begins with a standard backbone (e.g., HRNet) to generate initial feature maps. To enhance these features, a Semantic Interaction Convolution (SIC) module is introduced. This module groups feature maps that share similar visual patterns, allowing the network to distinguish between different joint types more explicitly before structural processing begins.

Stage 2: Adaptive Pose Hypergraph Neural Network (AP-HNN)

In the second stage, the model moves beyond standard graph edges—which connect only two nodes—to hyperedges, which can connect multiple nodes simultaneously. This allows for the capture of high-order group relationships (e.g., the coordination between a hand, foot, and torso during a complex stretch).

The AD-HNN architecture, showing the two-stage process from initial pose estimation to adaptive hypergraph learning.

Crucially, the hypergraph is adaptive. Rather than using a hard-coded skeleton, the AP-HNN employs an attention-like mechanism to update its incidence matrix during training. By calculating the semantic relevance between joints and hyperedges, the network learns the most plausible body structure for a specific pose in real-time. This is particularly effective for inferring the position of occluded joints based on their high-order relationships with visible parts of the body.

Key Insights: Dynamic Topology

The core innovation of AD-HNN is the transition from a bottlenecked, fixed skeletal map to a dynamic topology. By allowing the model to decide which joints are most related in a specific frame, it can accurately handle extreme poses where standard kinematic priors typically fail.

Comparison of initial pose structures versus the refined structures after adaptive adjustment, demonstrating the model's ability to handle complex poses.

Experimental Results

The AD-HNN was evaluated against major benchmarks, achieving state-of-the-art (SOTA) performance:

  • MPII Human Pose: Achieved a 92.4% [email protected] score, outperforming strong baselines like HRNet (92.3%).
  • CrowdPose: Reached 70.8 mAP, notably surpassing models specifically designed for occlusion like OPEC (70.6 mAP).
  • MS-COCO: On the test-dev set, it achieved 76.6 mAP, a 1.1% improvement over previous HRNet results.

Ablation studies confirmed that the adaptive nature of the hypergraph provided a ~1.0% mAP boost over static hypergraph configurations, validating the importance of dynamic structural learning.

Conclusion

AD-HNN demonstrates that high-order, adaptive relationships are essential for robust multi-person pose estimation. While currently focused on top-down estimation, future work will explore integrating these adaptive hypergraphs into bottom-up pipelines to improve efficiency for real-time applications involving large numbers of people.

Summary updated

BibTeX
@inproceedings{xu2022adaptive,
  author = {Xixia Xu and Qi Zou and Xue Lin},
  title = {{Adaptive Hypergraph Neural Network for Multi-Person Pose Estimation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i3.20201},
}
Figure