Camera Distance-Aware Top-Down Approach for 3D Multi-Person Pose Estimation From a Single RGB Image

Gyeongsik Moon, Ju Yong Chang, Kyoung Mu Lee

Abstract

Although significant improvement has been achieved recently in 3D human pose estimation, most of the previous methods only treat a single-person case. In this work, we firstly propose a fully learning-based, camera distance-aware top-down approach for 3D multi-person pose estimation from a single RGB image. The pipeline of the proposed system consists of human detection, absolute 3D human root localization, and root-relative 3D single-person pose estimation modules. Our system achieves comparable results with the state-of-the-art 3D single-person pose estimation models without any ground truth information and significantly outperforms previous 3D multi-person pose estimation methods on publicly available datasets.

Introduction

While 2D pose estimation has reached a high level of maturity, extending these capabilities to 3D space—particularly for multiple individuals in a single RGB image—remains a significant challenge. Most existing 3D methods are "root-relative," providing the position of joints relative to a person's pelvis but failing to locate that person within the global camera coordinate system.

The primary obstacle is depth ambiguity: in a 2D projection, a small child near the camera can appear identical in size to an adult further away. Without determining the absolute distance (the "root" location), it is impossible to reconstruct a coherent 3D scene where multiple people interact in a shared space. This paper introduces a robust top-down framework designed to solve this by accurately estimating the absolute distance of individuals from the camera.

Methodology

The proposed system utilizes a three-stage top-down pipeline to transition from a 2D image to a 3D multi-person reconstruction:

  1. DetectNet: A standard human detector (such as Mask R-CNN) is employed to identify and crop individual humans from the input image.
  2. RootNet: This novel module estimates the absolute 3D coordinates $(x, y, Z)$ of the person's root (pelvis). It combines geometric priors with deep learning. Specifically, it calculates a geometric distance measure ($k$) based on the bounding box area and camera focal lengths, then applies a CNN-learned correction factor ($\gamma$) to adjust for variations in body size and pose (e.g., crouching or sitting).
  3. PoseNet: A state-of-the-art 3D single-person pose estimator calculates the 3D positions of all joints relative to the root.
The proposed three-stage pipeline: DetectNet for human detection, RootNet for absolute 3D root localization, and PoseNet for root-relative 3D pose estimation.

By combining the absolute root location from RootNet with the relative joint positions from PoseNet, the system achieves absolute 3D coordinates for every joint of every person in the scene.

Experimental Results

The model was evaluated on several benchmarks, including Human3.6M for single-person accuracy and MuPoTS-3D for multi-person outdoor scenarios.

Key Findings

  • Superior Absolute Accuracy: On the MuPoTS-3D dataset, the system achieved a 3DPCK_abs score of 31.5, a significant improvement over the previous state-of-the-art baseline of 9.6. This demonstrates the effectiveness of the RootNet in resolving depth ambiguity.
  • Geometry-Aided Learning: The authors found that using a geometric "starting point" ($k$) and learning only the deviation ($\gamma$) is more effective than attempting to estimate absolute depth purely through a neural network.
  • Camera Independence: By factoring in camera intrinsics during the calculation of $k$, RootNet is "camera distance-aware," allowing it to generalize across different camera setups without retraining.
  • Efficiency: The pipeline processes frames at approximately 0.141 seconds on a TitanX GPU, making it viable for near real-time applications.
Qualitative results of the proposed system on diverse multi-person scenes, showing both the 2D projection and the reconstructed 3D poses in a shared coordinate system.

Conclusion

This work presents a significant step forward in 3D multi-person pose estimation by successfully integrating absolute root localization into a top-down framework. While the system relies on the quality of the initial human detector and assumes an average body size for its geometric prior, the introduction of the learned correction factor provides remarkable robustness. Future work may involve integrating explicit depth cues from dense depth estimation models to further refine absolute localization accuracy.

Summary updated

Abstract

Although significant improvement has been achieved recently in 3D human pose estimation, most of the previous methods only treat a single-person case. In this work, we firstly propose a fully learning-based, camera distance-aware top-down approach for 3D multi-person pose estimation from a single RGB image. The pipeline of the proposed system consists of human detection, absolute 3D human root localization, and root-relative 3D single-person pose estimation modules. Our system achieves comparable results with the state-of-the-art 3D single-person pose estimation models without any ground truth information and significantly outperforms previous 3D multi-person pose estimation methods on publicly available datasets. The code is available in <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1,2</sup> .
BibTeX
@inproceedings{moon2019camera,
  author = {Gyeongsik Moon and Ju Yong Chang and Kyoung Mu Lee},
  title = {{Camera Distance-Aware Top-Down Approach for 3D Multi-Person Pose Estimation From a Single RGB Image}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=a7xEbkMhGqL},
}
Figure