Monocular 3D Human Pose Estimation for Sports Broadcasts using Partial Sports Field Registration

Tobias Baumgartner, Stefanie KlattWorkshops @ CVPR 2023 · DOI · arXiv

Abstract

Estimating 3D human motion from a single broadcast camera is notoriously difficult because 2D images lose depth information. This paper introduces a method that uses the known geometry of sports fields (like lane lines on a track) to calibrate cameras and "ray-cast" 2D poses into accurate 3D space. By incorporating scene-specific priors such as the sagittal plane of motion and limb consistency, the proposed approach significantly reduces errors in joint angles and positions compared to standard "black-box" deep learning models.

1. The Challenge of Monocular 3D Pose Estimation

Human Pose Estimation (HPE) has achieved high precision in 2D, but "lifting" those 2D points into 3D space from a single camera remains an "ill-posed" problem. A single 2D point in an image can represent an infinite number of points along a 3D ray extending from the camera. This leads to what the authors call the "Ambiguity Trap": two vastly different 3D poses can project to the exact same 2D pixel coordinates if the camera settings differ.

The Ambiguity Trap: Different 3D poses can result in identical 2D projections, making monocular estimation inherently ill-posed.

Furthermore, most current 3D HPE models are trained in controlled laboratory settings with cameras positioned 6–20 meters away. In sports broadcasts, cameras are often hundreds of meters away, utilizing powerful telephoto lenses. This creates a massive "out-of-distribution" problem where models fail to account for specific scene geometry. Additionally, obtaining ground truth 3D data for elite athletes during real competition is nearly impossible, as they cannot wear motion-capture suits during a race.

2. Methodology: Geometry-Injected 3D Lifting

The authors propose a "geometry-injected" approach that utilizes the physical constraints of the environment rather than relying solely on a neural network's estimation. Their pipeline consists of three main stages:

  1. Partial Sports Field Registration: The method uses parallel lane lines on a track to identify vanishing points. While these lines do not immediately provide a perfect 3D location, they narrow down the camera's possible orientation and position to a single degree of freedom.
  2. Joint Optimization: By assuming the camera height remains constant across a video sequence, the system resolves the final degree of freedom, achieving full camera calibration.
  3. 3D Lifting via Ray-Casting: Instead of letting a model guess depth, the system casts rays from the camera through detected 2D joints. To locate the joint on that ray, two constraints are applied:
  4. The Sagittal Plane: The athlete's plane of motion is determined by observing where their feet touch the ground over time.
  5. Limb Consistency: The model enforces that an athlete’s bone lengths remain constant across frames, pruning physically impossible 3D configurations.
System pipeline comparing standard monocular lifting with the proposed geometry-aware approach.

3. Evaluation and Results

To validate the method, the authors developed a high-fidelity synthetic benchmark using Unreal Engine 5 (UE5). This dataset includes over 10,000 images featuring 31 sequences with variations in athlete height and running styles, providing perfect 3D ground truth.

The proposed method significantly outperformed MeTRAbs, a state-of-the-art monocular 3D HPE model:

  • 3D Position Error: Reduced from 10.33 cm (MeTRAbs) to 6.41 cm, and further to 2.44 cm when cyclical motion context was added.
  • Knee Angle Accuracy: Critical for biomechanical analysis, the error dropped from 20.31° to 9.91° (and 2.87° with context).
  • Qualitative Consistency: Skeletons produced by the proposed method remained "locked" to the track, whereas standard methods often produced skeletons that appeared to drift or float relative to the ground.

4. Conclusion and Future Work

The paper demonstrates that geometry serves as a powerful prior for 3D reconstruction. While the current registration method is optimized for straightaways, future work aims to adapt it for curved track sections and complex field markings in sports like soccer. Additionally, while the method currently assumes a stationary camera (allowing for pan, tilt, and zoom), future iterations could integrate these geometric constraints directly into the loss functions of deep learning models for more flexible, differentiable camera registration.

Summary updated

BibTeX
@inproceedings{baumgartner2023monocular,
  author = {Tobias Baumgartner and Stefanie Klatt},
  title = {{Monocular 3D Human Pose Estimation for Sports Broadcasts using Partial Sports Field Registration}},
  booktitle = {Workshops @ CVPR},
  year = {2023},
  doi = {10.1109/cvprw59228.2023.00539},
}
Figure