TL;DR
This paper introduces OpenCapBench, a novel benchmark that bridges the gap between computer vision's pose estimation and biomechanics' analysis of human movement. They also present SynthPose, a method for fine-tuning pose estimation models on synthetic data to predict denser, more biomechanically relevant keypoints, leading to significantly improved joint angle accuracy.
What Problem Does This Paper Address?
Computer vision has made impressive strides in estimating human pose from images and videos. However, current pose estimation benchmarks primarily focus on geometric accuracy (e.g., how close predicted keypoints are to ground truth) using metrics like Mean Per Joint Position Error (MPJPE). This overlooks a critical aspect: biomechanical validity. Biomechanics requires that joint movements adhere to physiological constraints (e.g., a knee joint has limited degrees of freedom). Existing pose estimation models often predict sparse keypoints that are insufficient for accurate biomechanical analysis, leading to physiologically implausible movement estimations. This disconnect hinders the application of pose estimation in fields like healthcare and sports science, where understanding the quality and correctness of movement is paramount.
Methods
The core contribution is OpenCapBench, a comprehensive pipeline that automates the process of evaluating pose estimation models using biomechanical metrics.
- Dataset Integration: It leverages the OpenCap dataset, which provides synchronized multi-view video data and ground truth 3D marker data from a motion capture system.
- 2D Keypoint Extraction: Users can plug in their 2D pose estimation models (predicting standard COCO keypoints) or SMPL shape estimation models. For SMPL models, anatomical markers are projected from the predicted mesh.
- 3D Triangulation: Multi-view 2D keypoints are deterministically triangulated into 3D keypoints.
- Inverse Kinematics (IK): The 3D keypoints are fed into the OpenSim musculoskeletal modeling software using a biomechanically validated model (Rajagopal model) to compute joint angles.
- Evaluation Metrics: The primary metric is the Root Mean Squared Error (RMSE) of specific joint angles (e.g., Pelvic Tilt, Knee Flexion) compared to ground truth from motion capture.
The paper also introduces SynthPose, a method to address the sparsity of keypoints from standard pose estimation models.
- Synthetic Data Generation: It uses synthetic datasets (like BEDLAM, Infinity VisionFit, 3DPW) where SMPL meshes are generated. A subset of SMPL vertices, corresponding to anatomical markers, are extracted and projected onto images to create ground-truth annotations for new keypoints.
- Fine-tuning: A pre-trained 2D pose estimation model (e.g., HRNet, ViTPose) has its final layer swapped to predict the new, denser set of keypoints. This model is then fine-tuned on the synthetically generated data. This approach leverages learned features from large datasets while adapting the model to predict biomechanically relevant keypoints.
Datasets
- OpenCap Dataset: Used as the primary dataset for the OpenCapBench benchmark. It includes multi-view video and motion capture data for various exercises (squats, sit-to-stand, walking).
- Synthetic Datasets for SynthPose Training:
- BEDLAM: Extensive synthetic dataset for training.
- Infinity VisionFit: Synthetic dataset with out-of-distribution samples and diverse exercise routines.
- 3DPW: "In-the-wild" dataset to bridge the sim-to-real gap, providing real-world examples.
- COCO: Included in the aggregated training set to mitigate sim-to-real distribution shift and prevent catastrophic forgetting.
- RICH Dataset: Used for evaluating SynthPose's ability to predict specific SMPL vertices (MoCap landmarks) on real-world data.
Key Insights and Ideas
- The "Pose Estimation vs. Biomechanics" Disconnect: Standard pose estimation metrics are insufficient for biomechanical analysis, leading to physiologically implausible results.
- Keypoint Sparsity is a Major Bottleneck: Current pose estimation models, even those predicting "whole body," often lack the specific keypoints needed for accurate joint angle calculations.
- Synthetic Data for Targeted Improvement: SynthPose demonstrates that fine-tuning on synthetically generated data with custom keypoint annotations can significantly improve biomechanical accuracy.
- OpenCapBench as a Unified Evaluation Platform: The benchmark provides a standardized way to assess pose estimation models not just on geometric accuracy but also on their biomechanical validity, fostering cross-disciplinary progress.
- Importance of Diverse Datasets: Ablation studies highlight that different datasets contribute uniquely to model performance on specific biomechanical metrics, underscoring the need for comprehensive training data.
Key Results
- SynthPose, when fine-tuned on synthetic data, reduces average joint angle RMSE by twofold and up to fourfold for certain biomechanically-relevant joints compared to standard pose estimation models.
- SynthPose significantly outperforms SMPL estimation-based methods on the RICH dataset, achieving 26% better PCK at 0.05 precision.
- When evaluated on OpenCapBench, SynthPose models (e.g., HRNet-W48, ViTPose-H) achieve 3-5x reduced RMSEs for key metrics like Pelvis Tilt, Hip Flexion, and Lumbar Extension compared to models predicting only COCO keypoints.
- SynthPose models also outperform the previous OpenCap method (which used an LSTM augmenter) on several joint angle metrics.
Limitations and Future Work
- Dataset Diversity: The OpenCap dataset, while biomechanically focused, could benefit from greater diversity in subjects, environmental settings, and activity ranges.
- Scope of Kinematics: The current benchmark primarily focuses on lower-body kinematics. Future work aims to include upper-body kinematics and limb assessments for a more holistic analysis.
- Integration of More Datasets: Incorporating additional MoCap-ground-truth datasets (like MoYo, PSU-TMM100) could extend the benchmark's applicability.
- Community Engagement: The authors encourage the community to explore other pose estimation tasks (3D, temporal) and different keypoint subsets with OpenCapBench.
Summary updated