Introduction
Estimating 3D human poses in large-scale, real-world environments—often referred to as "in the wild"—presents significant challenges. Traditional RGB-only methods frequently struggle with depth ambiguity and severe occlusions, where one subject may completely block another from a camera's view. Conversely, LiDAR-only approaches, while providing precise depth information, suffer from extreme data sparsity at long ranges. A person far from the sensor might be represented by only a few points, making skeletal reconstruction nearly impossible. Furthermore, the "annotation bottleneck" remains a major hurdle; manually labeling 3D joints for thousands of frames in diverse environments is prohibitively expensive.
LiCamPose addresses these challenges by fusing multi-view RGB camera data with sparse LiDAR point clouds into a unified 3D volumetric space. By leveraging a custom synthetic data generator, SyncHuman, and an entropy-guided unsupervised domain adaptation (UDA) strategy, the system achieves state-of-the-art accuracy without requiring manually labeled 3D pose annotations for the target environment.
Methodology: Volumetric Fusion and Adaptation
LiCamPose adopts a "top-down" volumetric fusion approach to reconcile the disparate data types of RGB and LiDAR.
Detection and Voxelization
The process begins with individual detection using a point-cloud-based detector, such as PointPillars. Once a person is detected, a 3D voxel grid—a cubic volume of spatial cells—is constructed around them. This grid serves as the common ground for multi-modal fusion.
Multi-Modal Fusion
The fusion happens at the voxel level:
- LiDAR Integration: Sparse point cloud data is mapped directly into the 3D grid, providing high-precision depth cues.
- RGB Integration: 2D pose heatmaps are generated for each available camera view. These heatmaps are then "back-projected" into the same 3D voxel grid, enriching the spatial volume with high-resolution semantic information.
V2V-Net and Estimation
A 3D convolutional neural network, V2V-Net, processes the fused volume. By operating in 3D space, the network can naturally resolve spatial conflicts and leverage the complementary strengths of both sensors to estimate final 3D joint locations.
Unsupervised Domain Adaptation (UDA)
To eliminate the need for real-world manual labels, LiCamPose is first trained on perfectly labeled synthetic data from the SyncHuman generator. It is then adapted to real-world target domains using two key mechanisms:
- Entropy-Guided Pseudo-labels: The model assesses the uncertainty (entropy) of its own predictions. High-confidence predictions (low entropy) are used as "pseudo-labels" to refine the model in the new environment.
- Human Priors: An unsupervised loss function penalizes anatomically impossible results, such as asymmetric limb lengths or unnatural joint angles, ensuring the estimated poses remain physically plausible.
Datasets and Evaluation
The efficacy of LiCamPose was evaluated across several environments, including the standard CMU Panoptic Studio and MVOR datasets. Crucially, the authors introduced the BasketBall dataset, a challenging real-world scenario captured with a four-view LiDAR-camera system, and SyncHuman, a Unity-based synthetic generator that simulates realistic LiDAR scanning patterns (e.g., Livox Mid-40).
Key Results
- Accuracy: On the synthetic BasketBallSync dataset, LiCamPose achieved a Mean Per-Joint Position Error (MPJPE) of 31.84 mm, significantly outperforming RGB-only voxel methods (40.80 mm).
- Unsupervised Performance: In large-scale real scenes, the unsupervised version of LiCamPose achieved an MPJPE of 72.92 mm, surpassing some fully supervised RGB-only methods.
- Robustness: Qualitative analysis demonstrates that the inclusion of LiDAR depth information allows the model to maintain accuracy even during severe occlusions or when 2D camera-based detectors fail.
Discussion and Future Work
The success of LiCamPose highlights the power of 3D volumetric fusion for multi-modal sensor data. By preserving spatial geometry, the model effectively reconciles the high-resolution texture of RGB with the precise depth of LiDAR. Furthermore, the strong correlation between mathematical entropy and physical rationality provides a robust mechanism for unsupervised learning.
However, limitations remain. The current model is "single-timestamp," processing frames in isolation without temporal tracking. Future work will focus on integrating temporal sequences to smooth jitter and handle even more extreme occlusions. Additionally, while the current simulation is tuned for specific LiDAR patterns, expanding SyncHuman to support more diverse sensor types and scenarios will further enhance the generalizability of the LiCamPose framework.
Summary updated