OpenCapBench: A Benchmark to Bridge Pose Estimation and Biomechanics

Yoni Gozlan, Antoine Falisse, Scott D. Uhlrich, Anthony A. Gatti, Michael Black, Akshay Chaudhari

TL;DR

This paper introduces OpenCapBench, a novel benchmark that bridges the gap between computer vision's pose estimation and biomechanics' analysis of human movement. It also presents SynthPose, a method that leverages synthetic data to train pose estimation models to predict denser keypoints, leading to more accurate biomechanical analysis.

What Problem Does This Paper Address?

Computer vision's pose estimation models have become increasingly sophisticated, but their evaluation often relies on metrics like Mean Per Joint Position Error (MPJPE) or Percentage of Correct Keypoints (PCK). While useful for visual accuracy, these metrics don't capture the physiological correctness or kinematic plausibility of human movement, which are crucial for biomechanics. Biomechanics, on the other hand, traditionally requires expensive motion capture setups and expert personnel. This disconnect means that pose estimation models, despite their potential, are not adequately evaluated for their real-world biomechanical applications, and biomechanics research lacks accessible, scalable tools for analyzing human movement from everyday video.

Methods

The core of this work is OpenCapBench, a comprehensive pipeline designed to evaluate pose estimation models under biomechanical constraints. It integrates with OpenSim, an open-source musculoskeletal modeling software, to derive kinematic metrics like joint angles. The pipeline takes 2D pose estimations, triangulates them to 3D, and then uses inverse kinematics within OpenSim to compute joint angles.

To address the sparsity of keypoints in current pose estimation models, the authors introduce SynthPose. This method uses synthetic data to fine-tune pre-trained 2D pose estimation models. The key idea is to train these models to predict a denser set of keypoints, derived from SMPL mesh vertices, which are more informative for kinematic analysis. SynthPose involves an automated labeling pipeline that projects SMPL vertices onto synthetic images to create ground-truth annotations for arbitrary keypoint sets.

Datasets

  • OpenCap Dataset: This is the primary dataset used for evaluation within OpenCapBench. It's a biomechanically-focused motion capture dataset containing multi-view video of subjects performing various exercises (squats, sit-to-stand, drop jumps, walking). It includes synchronized high-speed camera data and 3D marker data from an eight-camera MoCap system. Ground truth joint angles are derived from this MoCap data using OpenSim.
  • Synthetic Datasets for Training SynthPose:
    • BEDLAM Dataset: A large-scale synthetic dataset for training.
  • Infinity VisionFit: Synthetic data featuring diverse exercise routines and out-of-distribution samples.
  • 3DPW Dataset: Real-world "in-the-wild" data to help bridge the sim-to-real gap, particularly for subjects wearing shoes.
  • COCO Dataset: Included to ensure models can also output standard COCO keypoints and to mitigate sim-to-real distribution shift.

Key Insights and Ideas

  • Current pose estimation metrics are insufficient for biomechanics: Standard metrics like MPJPE and PCK don't capture the physiological realism of movement, leading to models that might look visually plausible but produce biomechanically incorrect joint angles.
  • Denser keypoints are crucial for accurate biomechanics: The paper demonstrates that current sparse keypoint predictions are too limited for precise kinematic analysis. Predicting a richer set of keypoints is essential.
  • Synthetic data is a powerful tool for specialized training: SynthPose effectively leverages synthetic data to train models for specific, biomechanically relevant tasks, overcoming limitations of real-world data annotation and diversity.
  • Bridging CV and Biomechanics is mutually beneficial: This work highlights how computer vision can provide accessible tools for biomechanics, while biomechanical constraints can lead to more robust and generalizable computer vision models.

Key Results

  • SynthPose, when fine-tuned on synthetic data, reduces average joint angle RMSE by half compared to models using sparse keypoints. For certain biomechanically relevant joints, the reduction is up to fourfold.
  • On the RICH dataset, SynthPose significantly outperforms state-of-the-art SMPL mesh prediction models by 26% in PCK at 0.05 precision.
  • The ablation studies on OpenCapBench show that specific datasets (like Infinity VisionFit) are crucial for improving predictions on particular anatomical features, offering nuanced insights not visible with traditional benchmarks.
  • The paper demonstrates that using a subset of SMPL vertices as keypoints for inverse kinematics leads to clear improvements across various joint angle metrics compared to using standard COCO keypoints.

Limitations and Future Work

  • Dataset Diversity: The authors acknowledge that the current dataset diversity within OpenCapBench is limited in terms of subject variety, environmental settings, and activity range. Future work will focus on integrating more datasets.
  • Upper Body Kinematics: The current benchmark primarily focuses on lower body kinematics. Expanding to include upper body and limb assessments would provide a more holistic view of human motion.
  • 3D Pose Estimation: While the focus is on 2D pose estimation, the authors suggest that SynthPose's principles could benefit monocular 3D models, which also suffer from keypoint scarcity.
  • Community Engagement: The authors encourage the community to extend OpenCapBench to other tasks like 3D keypoint estimation, temporal predictions, and experimenting with different keypoint subsets for inverse kinematics.

Summary updated

Abstract

Pose estimation has promised to impact healthcare by enabling more practical methods to quantify nuances of human movement and biomechanics. However, despite the inherent connection between pose estimation and biomechanics, these disciplines have largely remained disparate. For example, most current pose estimation benchmarks use metrics such as Mean Per Joint Position Error, Percentage of Correct Keypoints, or mean Average Precision to assess performance, without quantifying kinematic and physiological correctness - key aspects for biomechanics. To alleviate this challenge, we develop OpenCapBench to offer an easy-to-use unified benchmark to assess common tasks in human pose estimation, evaluated under physiological constraints. OpenCapBench computes consistent kinematic metrics through joints angles provided by an open-source musculoskeletal modeling software (OpenSim). Through OpenCapBench, we demonstrate that current pose estimation models use keypoints that are too sparse for accurate biomechanics analysis. To mitigate this challenge, we introduce SynthPose, a new approach that enables finetuning of pre-trained 2D human pose models to predict an arbitrarily denser set of keypoints for accurate kinematic analysis through the use of synthetic data. Incorporating such finetuning on synthetic data of prior models leads to twofold reduced joint angle errors. Moreover, OpenCapBench allows users to benchmark their own developed models on our clinically relevant cohort. Overall, OpenCapBench bridges the computer vision and biomechanics communities, aiming to drive simultaneous advances in both areas.
BibTeX
@inproceedings{gozlan2025opencapbench,
  author = {Yoni Gozlan and Antoine Falisse and Scott D. Uhlrich and Anthony A. Gatti and Michael Black and Akshay Chaudhari},
  title = {{OpenCapBench: A Benchmark to Bridge Pose Estimation and Biomechanics}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}

Loading related content...

Figure