TL;DR
This paper introduces a novel system for 3D human pose estimation that leverages a pair of vertically aligned 360° cameras. This setup offers complete scene coverage and portability, overcoming the limitations of traditional monocular (depth ambiguity) and multi-view (large footprint, restricted capture volume) systems, achieving accurate and temporally smooth pose estimates.
What Problem Does This Paper Address?
Estimating the 3D pose of a person from images is a long-standing challenge in computer vision with applications ranging from entertainment to medical fields. Traditional approaches fall into two main categories: monocular systems, which use a single camera but struggle with inherent depth ambiguity, and stereo/multi-view systems, which offer accurate depth but typically require multiple cameras, leading to expensive, bulky setups and limited capture volumes. The authors aim to bridge this gap by developing a system that combines the portability and ease of deployment of monocular systems with the depth accuracy of multi-view systems, all while providing complete scene coverage. They specifically address the challenge of achieving this with a compact, low-cost setup using readily available 360° cameras.
Methods
The proposed system employs a two-stage approach:
- Initial Skeletal Estimate:
- 3D Skeleton Representation: They first train a Principal Component Analysis (PCA) model on a dataset of 3D human poses to represent skeletons as a combination of parameters (pose, rotation, translation). This allows for expressing skeletons in a canonical form and facilitates optimization.
- 2D Joint Belief Maps: For each frame from the 360° cameras, they use OpenPose to extract 2D joint locations and approximate them as 2D Gaussians.
- Least Squares Optimization: A per-frame optimization is performed using Ceres Solver. This process fits the 3D skeleton to the 2D joint belief maps by minimizing a cost function that includes:
- Reprojection Error: How well the projected 3D skeleton joints match the detected 2D belief maps in both cameras.
- Motion Smoothness: Penalizing large changes in translation and rotation between consecutive frames.
- Size Constraint: Ensuring the skeleton's overall size is consistent.
- Joint Length Constraint: Enforcing realistic joint lengths based on the PCA model.
- Temporal Smoothing:
- Global Optimization: After obtaining initial per-frame estimates, a temporal smoothing stage is applied. This stage optimizes the 3D joint positions directly (not PCA parameters) by considering information from both previous and future frames.
- Refined Cost Function: The cost function here includes reprojection errors, but also introduces:
- Shift Error: Penalizing large movements of joints between consecutive frames.
- Alignment Error: Ensuring the current frame's joint positions are aligned with the temporal trajectory defined by the previous and next frames.
- Iterative Refinement: This smoothing process is repeated multiple times to further refine the pose estimates and improve temporal consistency.
Datasets
- Panoptic Studio Dataset: Used for training the PCA model and for quantitative evaluation. Specific scenes with vertically aligned cameras were selected.
- Synthetic Datasets: Two synthetic datasets were generated using Blender with a narrow camera baseline (30cm) for evaluation, allowing for comparison with different baseline configurations.
Key Insights and Ideas
- Vertical Baseline for 360° Cameras: The key insight is that a vertically aligned stereo pair of 360° cameras, despite having a narrow baseline, can effectively provide depth information for 3D pose estimation by leveraging the omnidirectional nature of the sensors. This is contrary to traditional stereo setups that rely on horizontal baselines.
- Combining Strengths: The system successfully merges the benefits of monocular (portability, complete scene coverage) and multi-view (depth accuracy) approaches.
- Temporal Consistency is Crucial: The paper emphasizes that temporal smoothing is not just an add-on but a critical component for achieving robust and smooth pose estimates, especially when dealing with potential per-frame inaccuracies.
- Exploiting Uncertainty: By using joint belief maps rather than absolute 2D joint positions in the initial optimization, the system can better handle uncertainty in the initial detection.
Key Results
- Quantitative Performance: The system achieved a Mean Per Joint Position Error (MPJPE) of 12.47cm on the Panoptic Studio dataset with a 30cm baseline.
- Improved Accuracy: On synthetic datasets with a narrow baseline, the proposed system outperformed monocular systems (Tome et al., Pavllo et al.) and showed competitive results against other stereo methods.
- Robustness to Projection Issues: Even with the inherent inaccuracies of projecting 360° images into an equirectangular format, the system demonstrated good performance.
- Temporal Smoothing Effectiveness: Qualitative results (Figure 7) show that the temporal smoothing significantly reduces jitter in the pose estimates, leading to much smoother motion trajectories compared to initial estimates or pure triangulation.
Limitations and Future Work
- Projection Inaccuracies: The paper acknowledges that projecting the Panoptic Studio dataset into 360° space introduces inexactness and potential errors, which can affect absolute accuracy.
- Camera Configuration: While the system works with vertically aligned cameras, it's noted that traditional stereo systems often rely on horizontal baselines for depth. The paper suggests future work could explore arbitrary vertical camera lens angles.
- Future Work: The authors suggest refining the system to operate with arbitrary vertical camera lens angles and potentially extending it to handle multiple people more robustly.
Summary updated