Are Existing Monocular Computer Vision-Based 3D Motion Capture Approaches Ready for Deployment? A Methodological Study on the Example of Alpine Skiing

Mirela Ostrek, Helge Rhodin, Pascal Fua, Erich Müller, Jörg Spörri

TL;DR

This paper investigates the readiness of monocular computer vision (MCV) for 3D motion capture in alpine skiing, comparing it to the gold standard stereophotogrammetry. They found that an MCV approach, using a single camera and deep learning, achieves accuracy and precision comparable to wearable sensors for key biomechanical metrics, suggesting its potential to revolutionize in-field sports analysis.

What Problem Does This Paper Address?

Collecting accurate 3D kinematic data for outdoor sports like alpine skiing is challenging. Traditional methods, such as marker-based stereophotogrammetry, are accurate but require complex multi-camera setups, extensive manual annotation, and are difficult to deploy in real-world field conditions. While wearable sensors offer a more practical alternative, they also have limitations. This paper explores whether monocular computer vision (MCV) approaches, leveraging deep learning and a single camera, can bridge this gap by providing sufficiently accurate and precise 3D motion capture for applied research in alpine skiing, specifically addressing the need for automated, less intrusive data collection.

Methods

The study employed a monocular computer vision (MCV) approach based on deep learning, specifically adapting the ResNet50 architecture. This method takes 2D images from a single camera and predicts the 3D pose of the skier, including joint locations and ski orientation. Crucially, the authors extended existing MCV methods to also output ski key-points, enabling the calculation of skiing-specific metrics. The training and testing involved a fully supervised pipeline, contrasting with some prior weakly-supervised methods. They implemented a six-fold cross-validation strategy to ensure robust evaluation. For comparison, they used video-based stereophotogrammetry as the "golden standard" reference method, which involved manual annotation and multi-view cameras.

Datasets

The study utilized a dataset collected from a field experiment involving six competitive alpine skiers performing four runs on a giant-slalom course. This dataset, captured by six synchronized high-definition video cameras, was used for both training and testing the MCV algorithm. The reference data for validation was also derived from this same experimental setup using stereophotogrammetry. No new benchmark dataset was introduced, but the existing data was meticulously processed and annotated for the study's purposes.

Key Insights and Ideas

  • MCV is surprisingly capable for complex sports: The study demonstrates that a single-camera MCV system, powered by deep learning, can achieve accuracy and precision in 3D motion capture for alpine skiing that is competitive with, and in some aspects superior to, established wearable sensor-based methods.
  • Automated annotation is a game-changer: The shift from manual annotation (stereophotogrammetry) to automated annotation (MCV) significantly reduces the practical barriers to deploying motion capture in real-world sports settings.
  • Ski-specific metrics are crucial: Simply estimating human pose isn't enough for sports analysis. The ability of the MCV to also infer ski orientation and calculate metrics like lean angle and drag area is vital for practical applications.
  • Camera perspective matters: While MCV aims for single-camera simplicity, the study highlights that certain camera angles (e.g., frontal-lateral) yield better results, suggesting that optimal camera placement can still be important for maximizing accuracy.

Key Results

  • 3D Joint Position Error: The normalized mean per joint position error (NMPJPE) for the MCV approach was 0.08 ± 0.01 m, which is comparable to or better than reported values for wearable sensor systems. The best performing cameras achieved an NMPJPE of 0.07 ± 0.01 m.
  • Knee and Hip Flexion Angles: Accuracy and precision for knee flexion were 0.4 ± 7.1° (7.2 ± 1.5°) for the outside leg and -0.2 ± 5.0° (6.7 ± 1.1°) for the inside leg. Hip flexion values were -0.4 ± 6.1° (4.4 ± 1.5°) and -0.7 ± 4.7° (3.7 ± 1.0°), respectively. These are considered acceptable for detecting relevant differences.
  • Skiing-Specific Metrics: Accuracy and precision for key skiing metrics were:
  • Relative center of mass position: 0.03 ± 0.01 m (0.01 ± 0.00 m)
    • Lean angle: -0.1 ± 3.8° (3.4 ± 0.9)
    • Center of mass to outside ankle distance: 0.01 ± 0.03 m (0.02 ± 0.01 m)
    • Fore/aft position: 0.01 ± 0.05 m (0.03 ± 0.01 m)
    • Drag area: 0.00 ± 0.01 m² (0.01 ± 0.00 m²) These values are deemed sufficient for detecting practically relevant differences in alpine skiing.

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the dataset used for training, while larger than some previous studies, could still be considered relatively small for deep learning. Augmenting existing data or collecting more diverse datasets could further improve accuracy.
  • Scale Ambiguity: The current MCV approach provides relative metrics due to the inherent scale ambiguity of monocular reconstruction. Estimating absolute position and velocity would require new algorithmic advancements.
  • Transferability: The performance of the algorithms might be limited to specific skiing situations (e.g., giant slalom). Transferring the approach to other disciplines like downhill skiing would necessitate application-specific training.
  • Future Work: The authors suggest that automating the manual process of centering and scale-normalizing input images would be a valuable next step. They also propose that the developed MCV approach could revolutionize kinematic data collection in sports science and medicine, especially for applications requiring large datasets for algorithm training.

Summary updated

Abstract

In this study, we compared a monocular computer vision (MCV)-based approach with the golden standard for collecting kinematic data on ski tracks (i.e., video-based stereophotogrammetry) and assessed its deployment readiness for answering applied research questions in the context of alpine skiing. The investigated MCV-based approach predicted the three-dimensional human pose and ski orientation based on the image data from a single camera. The data set used for training and testing the underlying deep nets originated from a field experiment with six competitive alpine skiers. The normalized mean per joint position error of the MVC-based approach was found to be 0.08 ± 0.01 m. Knee flexion showed an accuracy and precision (in parenthesis) of 0.4 ± 7.1° (7.2 ± 1.5°) for the outside leg, and −0.2 ± 5.0° (6.7 ± 1.1°) for the inside leg. For hip flexion, the corresponding values were −0.4 ± 6.1° (4.4° ± 1.5°) and −0.7 ± 4.7° (3.7 ± 1.0°), respectively. The accuracy and precision of skiing-related metrics were revealed to be 0.03 ± 0.01 m (0.01 ± 0.00 m) for relative center of mass position, −0.1 ± 3.8° (3.4 ± 0.9) for lean angle, 0.01 ± 0.03 m (0.02 ± 0.01 m) for center of mass to outside ankle distance, 0.01 ± 0.05 m (0.03 ± 0.01 m) for fore/aft position, and 0.00 ± 0.01 m2 (0.01 ± 0.00 m2) for drag area. Such magnitudes can be considered acceptable for detecting relevant differences in the context of alpine skiing.
BibTeX
@inproceedings{ostrek2019are,
  author = {Mirela Ostrek and Helge Rhodin and Pascal Fua and Erich Müller and Jörg Spörri},
  title = {{Are Existing Monocular Computer Vision-Based 3D Motion Capture Approaches Ready for Deployment? A Methodological Study on the Example of Alpine Skiing}},
  booktitle = {Sensors},
  year = {2019},
  url = {https://openreview.net/forum?id=Klr8VZXodx},
}
Figure