PointVoxel: A Simple and Effective Pipeline for Multi-View Multi-Modal 3D Human Pose Estimation

Zhiyu Pan, Zhicheng Zhong, Wenxuan Guo, Yifan Chen, Jianjiang Feng, Jie Zhou

Abstract

Several methods have been proposed to estimate 3D human pose from multi-view images, achieving satisfactory performance on public datasets collected under relatively simple conditions. However, there are limited approaches studying extracting 3D human skeletons from multimodal inputs, such as RGB and point cloud data. To address this gap, we introduce LiCamPose, a pipeline that integrates multi-view RGB and sparse point cloud information to estimate robust 3D human poses via single frame. We demonstrate the effectiveness of the volumetric architecture in combining these modalities. Furthermore, to circumvent the need for manually labeled 3D human pose annotations, we develop a synthetic dataset generator for pretraining and design an unsupervised domain adaptation strategy to train a 3D human pose estimator without manual annotations. To validate the generalization capability of our method, LiCamPose is evaluated on four datasets, including two public datasets, one synthetic dataset, and one challenging self-collected dataset named BasketBall, covering diverse scenarios. The results demonstrate that LiCamPose exhibits great generalization performance and significant application potential. The code, generator, and datasets will be made available upon acceptance of this paper.

Introduction

Estimating 3D human poses from multiple cameras has achieved significant success in controlled indoor environments. However, real-world "in-the-wild" scenarios—such as outdoor sports or crowded public spaces—present substantial challenges, including severe occlusions, varying lighting conditions, and large scene scales.

While RGB cameras offer high-resolution semantic information, they lack direct depth perception. Conversely, LiDAR provides precise 3D spatial data but is often too sparse to resolve intricate human joints independently. Moreover, the lack of large-scale, manually labeled 3D human pose datasets for outdoor multi-modal setups creates a "data desert." LiCamPose addresses these challenges by fusing complementary modalities and bridging the simulation-to-reality gap through synthetic data and unsupervised learning.

Overview of the LiCamPose pipeline, illustrating the use of synthetic data (SyncHuman) for pretraining and domain adaptation on real-world datasets like BasketBall.

Methodology

The LiCamPose pipeline employs a top-down volumetric fusion architecture designed for robustness and scalability.

Detection and Localization

The system utilizes PointPillars, a standard LiDAR-based detector, to identify individuals in 3D space. This provides a precise bounding volume for each person, ensuring accurate localization even in large-scale environments where image-based detection might falter.

Volumetric Fusion

For each detected individual, a 3D voxel grid is constructed. The system populates these voxels by back-projecting 2D joint heatmaps derived from RGB cameras and integrating raw LiDAR point cloud data. This volumetric approach creates a unified feature space that preserves the geometric properties of both sensor modalities.

Unsupervised Domain Adaptation (UDA)

To eliminate the need for manual real-world labels, a two-stage training strategy is implemented:

  1. Pretraining: The model is initially trained on perfectly labeled synthetic data generated by SyncHuman, a custom Unity-based generator.
  2. Self-Supervised Refinement: On real-world target data, the model is refined using a combination of 2D projection consistency and human prior losses (constraining bone lengths and joint angles).
  3. Entropy Filtering: The entropy of predicted 3D heatmaps serves as a confidence metric. High-confidence predictions (low entropy) are utilized as pseudo-labels for further network training.
The LiCamPose architecture and training strategy, featuring volumetric fusion of RGB and point cloud data alongside unsupervised domain adaptation losses.

Experimental Evaluation

LiCamPose was evaluated across four diverse datasets: SyncHuman (synthetic), BasketBall (outdoor real-world), CMU Panoptic Studio (indoor benchmark), and MVOR (operating room environment).

Key Results

  • State-of-the-Art Accuracy: On the CMU Panoptic dataset, LiCamPose achieved a Mean Per-Joint Position Error (MPJPE) of 14.44mm, surpassing established methods like VoxelPose (14.91mm).
  • Large-Scale Robustness: In the "BasketBallSync" environment, LiCamPose (31.84mm) significantly outperformed RGB-only methods such as PlanePose (119.55mm), demonstrating the necessity of LiDAR for scaling.
  • Unsupervised Performance: Even without manual 3D labels on target datasets, the unsupervised variant of LiCamPose (22.00mm on Panoptic) outperformed several fully supervised RGB-only competitors.
  • Ablation Studies: Integrating LiDAR into an RGB-only volumetric model resulted in a nearly 22% reduction in error for large-scale scenarios.

Conclusion and Future Work

LiCamPose demonstrates that fusing LiDAR and camera data within a volumetric framework significantly enhances 3D human pose estimation in complex environments. The use of synthetic data and entropy-guided UDA effectively mitigates the lack of labeled real-world data. Future research will focus on incorporating temporal information to improve joint tracking during rapid motions and optimizing the computational efficiency of the volumetric architecture for real-time multi-person applications.

Summary updated

Abstract

Several methods have been proposed to estimate 3D human pose from multi-view images, achieving satisfactory performance on public datasets collected under relatively simple conditions. However, there are limited approaches studying extracting 3D human skeletons from multimodal inputs, such as RGB and point cloud data. To address this gap, we introduce LiCamPose, a pipeline that integrates multi-view RGB and sparse point cloud information to estimate robust 3D human poses via single frame. We demonstrate the effectiveness of the volumetric architecture in combining these modalities. Furthermore, to circumvent the need for manually labeled 3D human pose annotations, we develop a synthetic dataset generator for pretraining and design an unsupervised domain adaptation strategy to train a 3D human pose estimator without manual annotations. To validate the generalization capability of our method, LiCamPose is evaluated on four datasets, including two public datasets, one synthetic dataset, and one challenging self-collected dataset named BasketBall, covering diverse scenarios. The results demonstrate that LiCamPose exhibits great generalization performance and significant application potential. The code, generator, and datasets will be made available upon acceptance of this paper.
BibTeX
@misc{pan2023pointvoxel,
  author = {Zhiyu Pan and Zhicheng Zhong and Wenxuan Guo and Yifan Chen and Jianjiang Feng and Jie Zhou},
  title = {{PointVoxel: A Simple and Effective Pipeline for Multi-View Multi-Modal 3D Human Pose Estimation}},
  year = {2023},
  eprint = {2312.06409},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.06409},
}
Figure