Abstract
Monocular 3D Human Pose Estimation (HPE) is a challenging ill-posed problem due to inherent depth ambiguity. While unsupervised methods have shown promise for single-person scenarios, they often struggle with multi-person interactions where perspective distortions lead to incorrect scaling and spatial positioning. This paper introduces an unsupervised pipeline that reconstructs 3D human interactions from 2D poses alone. By predicting camera elevation and implementing rotation and ground-plane compensation, the model aligns multiple skeletons into a shared, coherent 3D coordinate system. Experimental results on the CHI3D dataset demonstrate that rotation compensation alone reduces position error by 23.4%, establishing a robust benchmark for unsupervised multi-person 3D lifting.
Introduction: The Challenge of Monocular Depth
Estimating 3D human poses from a single 2D image is inherently difficult because a single 2D projection can correspond to multiple 3D configurations. In unsupervised learning, researchers typically solve this for a single person by "root-centering" the pose—fixing the pelvis at a set distance from the camera. However, this approach breaks down during human-to-human interactions. When two people are at different depths, the person further away appears smaller and often seems to "float" relative to the foreground figure. This paper addresses the need for a shared 3D coordinate system that maintains the natural scale and spatial relationship between interacting individuals without relying on 3D ground truth during training.
Methodology: From 2D Skeletons to 3D Scenes
The proposed method utilizes a multi-stage pipeline to transform independent 2D keypoints into a unified 3D reconstruction:
- Independent Lifting: The system first "lifts" each person's 2D pose to 3D independently using a neural network based on the LInKs algorithm. Initially, both individuals are placed at a fixed distance from the camera.
- Elevation Prediction: The core innovation involves predicting the camera's elevation angle ($\theta$) relative to each person's pelvis. This allows the model to understand the camera's tilt, which is a primary source of perspective error.
- Rotation and Offset Compensation: Using the predicted elevation, the system calculates vertical offsets and applies a rotation matrix to "level" the poses with a common ground plane.
- Ground Plane Scaling: Finally, the poses are scaled to ensure their feet align on the same 3D ground, preventing unrealistic hovering and ensuring correct relative sizing.
By focusing on 2D keypoints rather than full RGB images, the model remains lightweight and avoids the computational overhead of heavy volumetric models, making it suitable for potential real-time applications.
Experimental Results and Evaluation
The researchers evaluated their approach using the CHI3D (Close Human Interactions) dataset, which contains high-quality motion capture data of pairs engaging in activities like hugging, pushing, or grabbing. To measure performance in multi-person contexts, they introduced three new metrics: Scale Error (SE), Translation Error (TE), and Root Displacement Error (RDE).
The inclusion of elevation and rotation compensation proved critical. Rotation compensation alone reduced the Position Error (PA-MPJPE) by 23.4%, with the full model achieving a final accuracy of 149.4 mm. Qualitatively, the results demonstrate that background figures are no longer disconnected or incorrectly scaled, but instead interact naturally within the reconstructed 3D space.
Discussion and Future Directions
While the method significantly improves multi-person 3D lifting, it remains dependent on the accuracy of the initial 2D pelvis detection. Errors in the 2D input propagate through the elevation and scaling logic. Additionally, discrepancies between 2D annotations and 3D ground truth in existing datasets pose a challenge for training.
Future work aims to incorporate a contact detector. By identifying specific points of physical contact (e.g., a hand on a shoulder), the model could use these as geometric anchors to further refine the relative scaling and displacement of the 3D skeletons, leading to even more precise interaction modeling.
Summary updated