OpenCapBench: A Benchmark to Bridge Pose Estimation and Biomechanics

Yoni Gozlan, Antoine Falisse, Scott Uhlrich, Anthony Gatti, Michael Black, Jennifer Hicks, Scott Delp, Akshay Chaudhari

TL;DR

This paper introduces OpenCapBench, a novel benchmark that bridges the gap between computer vision's pose estimation and biomechanics' analysis of human movement. They also present SynthPose, a method for fine-tuning pose estimation models on synthetic data to predict denser, more biomechanically relevant keypoints, leading to significantly improved joint angle accuracy.

What Problem Does This Paper Address?

Computer vision has made impressive strides in estimating human pose from images and videos. However, current pose estimation benchmarks primarily focus on geometric accuracy (e.g., how close predicted keypoints are to ground truth) using metrics like Mean Per Joint Position Error (MPJPE). This overlooks a critical aspect: biomechanical validity. Biomechanics requires that joint movements adhere to physiological constraints (e.g., a knee joint has limited degrees of freedom). Existing pose estimation models often predict sparse keypoints that are insufficient for accurate biomechanical analysis, leading to physiologically implausible movement estimations. This disconnect hinders the application of pose estimation in fields like healthcare and sports science, where understanding the quality and correctness of movement is paramount.

Methods

The core contribution is OpenCapBench, a comprehensive pipeline that automates the process of evaluating pose estimation models using biomechanical metrics.

  1. Dataset Integration: It leverages the OpenCap dataset, which provides synchronized multi-view video data and ground truth 3D marker data from a motion capture system.
  2. 2D Keypoint Extraction: Users can plug in their 2D pose estimation models (predicting standard COCO keypoints) or SMPL shape estimation models. For SMPL models, anatomical markers are projected from the predicted mesh.
  3. 3D Triangulation: Multi-view 2D keypoints are deterministically triangulated into 3D keypoints.
  4. Inverse Kinematics (IK): The 3D keypoints are fed into the OpenSim musculoskeletal modeling software using a biomechanically validated model (Rajagopal model) to compute joint angles.
  5. Evaluation Metrics: The primary metric is the Root Mean Squared Error (RMSE) of specific joint angles (e.g., Pelvic Tilt, Knee Flexion) compared to ground truth from motion capture.

The paper also introduces SynthPose, a method to address the sparsity of keypoints from standard pose estimation models.

  1. Synthetic Data Generation: It uses synthetic datasets (like BEDLAM, Infinity VisionFit, 3DPW) where SMPL meshes are generated. A subset of SMPL vertices, corresponding to anatomical markers, are extracted and projected onto images to create ground-truth annotations for new keypoints.
  2. Fine-tuning: A pre-trained 2D pose estimation model (e.g., HRNet, ViTPose) has its final layer swapped to predict the new, denser set of keypoints. This model is then fine-tuned on the synthetically generated data. This approach leverages learned features from large datasets while adapting the model to predict biomechanically relevant keypoints.

Datasets

  • OpenCap Dataset: Used as the primary dataset for the OpenCapBench benchmark. It includes multi-view video and motion capture data for various exercises (squats, sit-to-stand, walking).
  • Synthetic Datasets for SynthPose Training:
    • BEDLAM: Extensive synthetic dataset for training.
  • Infinity VisionFit: Synthetic dataset with out-of-distribution samples and diverse exercise routines.
  • 3DPW: "In-the-wild" dataset to bridge the sim-to-real gap, providing real-world examples.
  • COCO: Included in the aggregated training set to mitigate sim-to-real distribution shift and prevent catastrophic forgetting.
  • RICH Dataset: Used for evaluating SynthPose's ability to predict specific SMPL vertices (MoCap landmarks) on real-world data.

Key Insights and Ideas

  • The "Pose Estimation vs. Biomechanics" Disconnect: Standard pose estimation metrics are insufficient for biomechanical analysis, leading to physiologically implausible results.
  • Keypoint Sparsity is a Major Bottleneck: Current pose estimation models, even those predicting "whole body," often lack the specific keypoints needed for accurate joint angle calculations.
  • Synthetic Data for Targeted Improvement: SynthPose demonstrates that fine-tuning on synthetically generated data with custom keypoint annotations can significantly improve biomechanical accuracy.
  • OpenCapBench as a Unified Evaluation Platform: The benchmark provides a standardized way to assess pose estimation models not just on geometric accuracy but also on their biomechanical validity, fostering cross-disciplinary progress.
  • Importance of Diverse Datasets: Ablation studies highlight that different datasets contribute uniquely to model performance on specific biomechanical metrics, underscoring the need for comprehensive training data.

Key Results

  • SynthPose, when fine-tuned on synthetic data, reduces average joint angle RMSE by twofold and up to fourfold for certain biomechanically-relevant joints compared to standard pose estimation models.
  • SynthPose significantly outperforms SMPL estimation-based methods on the RICH dataset, achieving 26% better PCK at 0.05 precision.
  • When evaluated on OpenCapBench, SynthPose models (e.g., HRNet-W48, ViTPose-H) achieve 3-5x reduced RMSEs for key metrics like Pelvis Tilt, Hip Flexion, and Lumbar Extension compared to models predicting only COCO keypoints.
  • SynthPose models also outperform the previous OpenCap method (which used an LSTM augmenter) on several joint angle metrics.

Limitations and Future Work

  • Dataset Diversity: The OpenCap dataset, while biomechanically focused, could benefit from greater diversity in subjects, environmental settings, and activity ranges.
  • Scope of Kinematics: The current benchmark primarily focuses on lower-body kinematics. Future work aims to include upper-body kinematics and limb assessments for a more holistic analysis.
  • Integration of More Datasets: Incorporating additional MoCap-ground-truth datasets (like MoYo, PSU-TMM100) could extend the benchmark's applicability.
  • Community Engagement: The authors encourage the community to explore other pose estimation tasks (3D, temporal) and different keypoint subsets with OpenCapBench.

Summary updated

Abstract

Pose estimation has promised to impact healthcare by enabling more practical methods to quantify nuances of human movement and biomechanics. However, despite the inherent connection between pose estimation and biomechanics, these disciplines have largely remained disparate. For example, most current pose estimation benchmarks use metrics such as Mean Per Joint Position Error, Percentage of Correct Keypoints, or mean Average Precision to assess performance, without quantifying kinematic and physiological correctness - key aspects for biomechanics. To alleviate this challenge, we develop OpenCapBench to offer an easy-to-use unified benchmark to assess common tasks in human pose estimation, evaluated under physiological constraints. OpenCapBench computes consistent kinematic metrics through joints angles provided by an open-source musculoskeletal modeling software (OpenSim). Through OpenCapBench, we demonstrate that current pose estimation models use keypoints that are too sparse for accurate biomechanics analysis. To mitigate this challenge, we introduce SynthPose, a new approach that enables finetuning of pre-trained 2D human pose models to predict an arbitrarily denser set of keypoints for accurate kinematic analysis through the use of synthetic data. Incorporating such finetuning on synthetic data of prior models leads to twofold reduced joint angle errors. Moreover, OpenCapBench allows users to benchmark their own developed models on our clinically relevant cohort. Overall, OpenCapBench bridges the computer vision and biomechanics communities, aiming to drive simultaneous advances in both areas.
BibTeX
@inproceedings{gozlan2025opencapbench,
  author = {Yoni Gozlan and Antoine Falisse and Scott Uhlrich and Anthony Gatti and Michael Black and Jennifer Hicks and Scott Delp and Akshay Chaudhari},
  title = {{OpenCapBench: A Benchmark to Bridge Pose Estimation and Biomechanics}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure