Inter-image Contrastive Consistency for Multi-Person Pose Estimation

Xixia Xu, Yingguo Gao, Xingjia Pan, Ke Yan, Xiaoyu Chen, Qi ZouAAAI 2023 · DOI

Abstract

Multi-person pose estimation (MPPE) often struggles with occlusions and appearance variations. We present ICON (Inter-image CONtrastive consistency), a framework that enhances pose estimation by enforcing feature consistency for human joints across diverse images. By leveraging inter-image relationships through Single Keypoint Contrastive Consistency (SKCC) and Pair Relation Contrastive Consistency (PRCC), ICON improves robustness to extreme postures and crowded scenes without increasing inference-time complexity.

Introduction

Multi-person pose estimation involves identifying the spatial coordinates of human joints (keypoints) for all individuals in an image. Despite advancements in deep learning, current models frequently fail in complex scenarios characterized by appearance variance (e.g., lighting, clothing) and heavy occlusions. Traditional approaches primarily focus on intra-image spatial relationships—how joints within a single person relate to one another. However, this often fails to capture a universal representation of joint categories, leading to localization jitter or missed detections in challenging poses.

Methodology

ICON addresses these limitations by introducing contrastive learning constraints that operate across the entire dataset rather than within a single image.

Single Keypoint Contrastive Consistency (SKCC)

SKCC forces the model to map identical keypoint types (e.g., all "left ankles") from different images to similar regions in a high-dimensional feature space. Simultaneously, it pushes different keypoint types (e.g., ankle vs. wrist) apart. This sharpens the model's categorical understanding of each joint, making it more resilient to varying backgrounds and lighting.

Pair Relation Contrastive Consistency (PRCC)

To incorporate structural knowledge, PRCC enforces consistency in the relationships between adjacent joints, such as the vector from elbow to wrist. This provides relational guidance, allowing the model to infer the position of occluded joints based on their visible neighbors.

Hard Example Sampling

To optimize training, ICON employs Hard Example Sampling. The system identifies joints with high uncertainty or loss—typically those in crowded or occluded regions—and prioritizes them for contrastive loss calculation. Importantly, the representation head used for these calculations is discarded after training, ensuring that ICON improves accuracy with zero additional computational cost during inference.

Experimental Results

The effectiveness of ICON was evaluated on major benchmarks, including MS-COCO, MPII Human Pose, and CrowdPose.

Qualitative comparison of pose estimation results across three scenes. (a) Baseline models often struggle with occlusions and crowded scenes (red boxes). (b) Adding SKCC improves joint localization (pink boxes). (c) The full ICON model (+SKCC+PRCC) provides the most robust results by incorporating structural consistency (orange boxes).

On MS-COCO, ICON improved the Mean Average Precision (mAP) of HRNet and ResNet backbones by 1.6% and 1.7%, respectively. In the more challenging CrowdPose dataset, it achieved a 1.3% mAP improvement over the SBN baseline. Diagnostic analysis confirmed that ICON significantly reduces "jitter" and "swap" errors, where keypoints are confused between different individuals.

Discussion and Conclusion

A key insight from this work is that inter-image consistency is more effective than intra-image relations for learning robust, category-specific features. PRCC acts as a vital structural corrector, mitigating "inversion" errors (e.g., confusing left and right limbs). Furthermore, ICON demonstrates remarkable data efficiency; in semi-supervised settings, models trained with ICON on only 30% of labeled data performed comparably to standard models using 100% of labels.

While ICON provides a significant performance boost, future work could explore dynamic or higher-order joint relations beyond the current predefined skeleton. Nevertheless, the principle of inter-image consistency offers a powerful, inference-free method for improving dense prediction tasks across various domains.

Summary updated

BibTeX
@inproceedings{xu2023inter,
  author = {Xixia Xu and Yingguo Gao and Xingjia Pan and Ke Yan and Xiaoyu Chen and Qi Zou},
  title = {{Inter-image Contrastive Consistency for Multi-Person Pose Estimation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i3.25410},
}
Figure