I²R-Net: Intra- and Inter-Human Relation Network for Multi-Person Pose Estimation

Yiwei Ding, Wenjin Deng, Yinglin Zheng, Pengfei Liu, Meihong Wang, Xuan Cheng, Jianmin Bao, Dong Chen, Ming Zeng

Abstract

In this paper, we present the Intra- and Inter-Human Relation Networks I²R-Net for Multi-Person Pose Estimation. It involves two basic modules. First, the Intra-Human Relation Module operates on a single person and aims to capture Intra-Human dependencies. Second, the Inter-Human Relation Module considers the relation between multiple instances and focuses on capturing Inter-Human interactions. The Inter-Human Relation Module can be designed very lightweight by reducing the resolution of feature map, yet learn useful relation information to significantly boost the performance of the Intra-Human Relation Module. Even without bells and whistles, our method can compete or outperform current competition winners. We conduct extensive experiments on COCO, CrowdPose, and OCHuman datasets. The results demonstrate that the proposed model surpasses all the state-of-the-art methods. Concretely, the proposed method achieves 77.4% AP on CrowPose dataset and 67.8% AP on OCHuman dataset respectively, outperforming existing methods by a large margin.

Introduction

Multi-person pose estimation (MPPE) involves identifying anatomical keypoints for every individual in an image. While current top-down and bottom-up approaches perform well in standard scenarios, they often falter in crowded scenes where people overlap and occlusions are frequent. Top-down methods typically treat individuals in isolation, missing contextual cues from nearby people. Bottom-up methods struggle with limb assignment in dense environments.

Visualizing intra-human (green) and inter-human (yellow) dependencies in complex scenes.

I²R-Net addresses these challenges by explicitly modeling both the spatial dependencies within a single body (Intra-Human) and the complex interactions between different people (Inter-Human). This dual-relational approach allows the model to resolve ambiguities that traditional methods cannot.

Methods

The I²R-Net architecture consists of two primary stages designed to capture local and global human relations.

The overall architecture of I²R-Net, featuring Intra- and Inter-Human Relation Modules.

Intra-Human Relation Module

This module operates at high resolution on individual person patches. Its primary goal is to learn the structural dependencies of a single body, such as the relative positioning of limbs (e.g., where the wrist should be relative to the elbow). This stage is modular and can incorporate various existing backbones like HRNet or TransPose.

Inter-Human Relation Module

The core innovation of I²R-Net is the Inter-Human Relation Module. It processes feature maps from the first stage by downsampling them to a lower resolution and flattening them into tokens. These tokens are processed by a Transformer block using self-attention to correlate features across different individuals. This allows the network to use visible parts of one person to infer the occluded parts of another, effectively "denoising" ambiguous poses in crowds.

Key Insights

  • Efficiency of Global Relations: A key finding is that inter-human interactions can be effectively captured at low resolutions. This allows the Inter-Human module to be lightweight, minimizing the computational cost associated with Transformer blocks.
  • Contextual Inference: By looking at neighbors, the model can resolve ambiguities caused by occlusion. For instance, in a crowded scene, the visible limbs of one person provide the necessary context to place the hidden limbs of another.
  • Modularity: The framework is "plug-and-play," meaning it can enhance the performance of various base single-person pose estimators.

Experimental Results

The model was evaluated on COCO, CrowdPose, and OCHuman datasets, setting new benchmarks in performance:

  • CrowdPose: Achieved 77.4% AP, a significant 5.0% improvement over previous state-of-the-art models like HRFormer.
  • OCHuman: Reached 67.8% AP, surpassing TransPose-H by 5.5%.
  • COCO: Maintained high performance in non-crowded scenes with 77.3% AP, demonstrating that relational modeling is beneficial across diverse scenarios.

Conclusion

I²R-Net demonstrates that explicitly modeling inter-human relations is crucial for robust pose estimation in crowded and occluded environments. While the model still faces challenges in cases of extreme occlusion or weak correlation between individuals, it provides a powerful framework for contextual reasoning. Future work may involve extending these relational models to the temporal domain for video-based pose estimation.

Summary updated

Abstract

In this paper, we present the Intra- and Inter-Human Relation Networks I²R-Net for Multi-Person Pose Estimation. It involves two basic modules. First, the Intra-Human Relation Module operates on a single person and aims to capture Intra-Human dependencies. Second, the Inter-Human Relation Module considers the relation between multiple instances and focuses on capturing Inter-Human interactions. The Inter-Human Relation Module can be designed very lightweight by reducing the resolution of feature map, yet learn useful relation information to significantly boost the performance of the Intra-Human Relation Module. Even without bells and whistles, our method can compete or outperform current competition winners. We conduct extensive experiments on COCO, CrowdPose, and OCHuman datasets. The results demonstrate that the proposed model surpasses all the state-of-the-art methods. Concretely, the proposed method achieves 77.4% AP on CrowPose dataset and 67.8% AP on OCHuman dataset respectively, outperforming existing methods by a large margin. Additionally, the ablation study and visualization analysis also prove the effectiveness of our model.
BibTeX
@inproceedings{ding2022i,
  author = {Yiwei Ding and Wenjin Deng and Yinglin Zheng and Pengfei Liu and Meihong Wang and Xuan Cheng and Jianmin Bao and Dong Chen and Ming Zeng},
  title = {{I²R-Net: Intra- and Inter-Human Relation Network for Multi-Person Pose Estimation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=54TVnEvoUf},
}
Figure