Learning Spatial Context with Graph Neural Network for Multi-Person Pose Grouping

Jiahao Lin, Gim Hee Lee

Abstract

Multi-person pose estimation often struggles with grouping detected keypoints in complex scenes. This paper presents a novel bottom-up framework that leverages Graph Neural Networks (GNNs) to learn spatial context and geometric constraints. By fusing these learned geometric affinities with traditional visual appearance features, the method achieves significant improvements in grouping accuracy, particularly in cases of overlapping individuals and varying scales. The proposed approach demonstrates state-of-the-art performance on major benchmarks like COCO and exhibits strong generalization capabilities.

Introduction

In the field of multi-person pose estimation, "bottom-up" approaches have gained popularity due to their computational efficiency. These methods detect all body joints (keypoints) in an image simultaneously and subsequently group them into individual person instances. However, most existing grouping techniques rely almost exclusively on visual appearance embeddings. This reliance introduces several challenges: visual features are sensitive to environmental factors like lighting and background clutter; embeddings often struggle with scale variance when people appear at different sizes; and models frequently ignore inherent human anatomical constraints.

To address these issues, this research introduces a framework that teaches models to reason about the spatial relationships between joints using a Graph Neural Network. By incorporating "geometry-aware" association, the model can leverage the rules of human anatomy to improve grouping robustness.

Methodology

The authors formulate the grouping task as a graph partitioning problem. Every detected keypoint is treated as a node in a fully connected graph, and the objective is to determine which edges connect joints belonging to the same person.

Overview of the proposed multi-person pose grouping framework, showing the parallel geometry and appearance branches.

The framework consists of two primary parallel branches:

  1. Geometry-aware Association GNN: This core innovation uses only $(x, y)$ coordinates and joint types as input. Through iterative message passing, each joint aggregates global context from all other detected joints, learning a geometric affinity score based on anatomical feasibility.
  2. Appearance-based Association: In parallel, a standard GNN processes visual feature patches from the image to compute an appearance affinity score, similar to traditional associative embedding methods.

The geometric and appearance scores are fused into a final affinity matrix. Finally, spectral clustering is applied to partition the graph into distinct clusters, each representing a single person instance. This dual-pathway approach allows the model to benefit from both visual cues and structural priors.

Experiments and Results

The model was rigorously evaluated on the COCO Keypoint Dataset and the MPII Human Pose Dataset.

Key Insights

  • The Power of Geometry: The geometry-aware branch alone, using no image data, achieved over 97% accuracy in edge classification. This confirms that spatial configuration is a highly discriminative feature for human pose grouping.
  • Global Reasoning: Unlike local methods, the GNN allows for global context. A joint's assignment is informed by its position relative to the entire set of detected keypoints.
  • Scale Robustness: By learning relative spatial relationships, the GNN is significantly more robust to scale variations than pure appearance-based methods.

Quantitative and Qualitative Performance

On the COCO dataset, the proposed method outperformed the baseline appearance-only approach by 2.2% AP on the validation set and 1.7% AP on the test-dev set. It achieved an overall 68.2% AP on COCO test-dev, which is competitive with state-of-the-art bottom-up methods like PersonLab.

Qualitative comparison demonstrating how the proposed method (bottom) correctly groups poses where appearance-only methods (top) fail due to overlap or unusual configurations.

Furthermore, when applied to the MPII dataset without fine-tuning, the model outperformed the baseline by 1.5% AP. This demonstrates that the learned geometric "rules" of the human body are universal and transfer well across different datasets.

Conclusion and Future Work

By integrating spatial context through Graph Neural Networks, this framework significantly enhances the accuracy and robustness of multi-person pose grouping. While the method currently depends on the quality of initial keypoint detections and involves some computational overhead during spectral clustering, it provides a powerful mechanism for incorporating anatomical priors into deep learning models. Future research will focus on optimizing the partitioning stage for real-time applications and improving robustness to heavy occlusions.

Summary updated

BibTeX
@inproceedings{lin2021learning,
  author = {Jiahao Lin and Gim Hee Lee},
  title = {{Learning Spatial Context with Graph Neural Network for Multi-Person Pose Grouping}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  doi = {10.1109/icra48506.2021.9561960},
}
Figure