Extraction of Positional Player Data From Broadcast Soccer Videos

Jonas Theiner, Wolfgang Gritz, Eric Müller-Budack, Robert Rein, Daniel Memmert, Ralph EwerthWACV 2022

TL;DR

This paper introduces a fully automated pipeline for extracting 2D player positions from broadcast soccer videos, a crucial but challenging task due to data scarcity and the complexity of video analysis. By integrating and evaluating individual modules for field registration, player detection, and team assignment, the authors provide a robust baseline and novel evaluation metrics for this domain.

What Problem Does This Paper Address?

Accurate player positional data is invaluable for in-depth soccer analysis, from scouting and performance evaluation to tactical strategy. However, obtaining this data is difficult. While professional tracking systems exist, they are often proprietary and expensive. Broadcast videos, on the other hand, are widely accessible but present significant challenges: varying camera angles, player occlusion, and the need to infer 3D positions from 2D images. Existing research often tackles individual sub-problems (like field registration or player detection) in isolation, with limited evaluation on real-world broadcast data and little insight into how errors propagate through a complete pipeline. This paper aims to bridge this gap by presenting a comprehensive, modular system and evaluating its performance rigorously.

Methods

The proposed pipeline is a modular system designed to extract player positions from broadcast soccer videos. The key stages are:

  • Shot Boundary and Shot Type Classification: This initial step identifies relevant segments of the video, specifically focusing on shots from the "main camera" which typically offers the most comprehensive view of the field. They use TransNet for boundary detection and then classify shot types to isolate the main camera feed.
  • Sports Field Registration: This is a critical step that determines a homography matrix to map image coordinates to a standardized 2D sports field coordinate system. The authors adapt Chen and Little's approach, which uses a pix2pix model for field segmentation and edge detection, followed by a Siamese CNN for matching against synthetic data to estimate camera parameters and an initial homography. This is then refined using the Lucas-Kanade algorithm.
  • Player Detection: The system employs a general-purpose object detection framework. While not explicitly detailing the chosen model in the main text, they compare their system's performance against fine-tuned Faster R-CNN and a specialized network (FootAndBall), ultimately selecting CenterTrack for its good generalizability.
  • Team Assignment: To differentiate between teams, the authors use a DBSCAN clustering algorithm on player bounding box features (specifically the upper torso area). This unsupervised approach leverages the color information of jerseys and avoids the need for labeled training data for this specific module.
  • Self-Verification: A novel self-verification step is introduced to discard frames where the sports field registration is likely erroneous. This is achieved by checking if projected player positions fall within a defined tolerance distance from the field boundaries.

Datasets

The paper utilizes several datasets for evaluation:

  • Training Data: For individual modules like shot boundary detection, shot type classification, and player detection, the authors leverage publicly available datasets and existing trained models. For sports field registration, they mention using a subset of their own data for manual annotation.
  • Evaluation Data: The core evaluation of the entire pipeline is performed on four distinct datasets: TV12 (2012, SD), TV14 (2014, HD), TC14 (HD, tactic-cam), and TV14-S. These datasets consist of synchronized broadcast video and ground-truth positional data from a calibrated multi-camera system, providing a realistic testbed. The tactic-cam data is noted as being particularly useful for analysis due to its continuous coverage.

Key Insights and Ideas

  • Modularity and Error Propagation: The paper emphasizes the interdependent nature of the pipeline modules. Errors in early stages, particularly sports field registration, can significantly impact downstream tasks like player position estimation.
  • Novel Evaluation Metrics: The authors introduce new metrics to address the complexities of evaluating player position estimation from broadcast video, such as accounting for partially visible players and the impact of outliers.
  • Generalizability of Sports Field Registration: The adapted Chen and Little approach shows promising generalizability across different camera parameters and environments, even without extensive fine-tuning on new datasets.
  • Unsupervised Team Assignment: The use of DBSCAN for team assignment is a practical approach that avoids the need for labeled team data, making the system more adaptable.
  • Importance of Main Camera Feed: Focusing on the "main camera" feed is a pragmatic choice that balances comprehensive field coverage with manageable data processing.

Key Results

  • Player Detection: CenterTrack demonstrated good generalizability across different datasets, outperforming other tested models in this regard.
  • Sports Field Registration: The adapted approach achieved competitive results, with a slight drop in performance on datasets with unknown camera parameters compared to the WC14 benchmark. However, it showed better adaptability than some other methods.
  • Team Assignment: The unsupervised DBSCAN approach achieved high accuracy (micro accuracy of 0.93 for team classes) with most errors occurring when assigning goalkeepers or referees.
  • Overall System Performance: The pipeline, after self-verification, provides promising results, with mean distances between estimated and ground-truth positions often within a few meters. The paper highlights that even small errors in sports field registration can lead to significant absolute errors in player positions.

Limitations and Future Work

  • Player Tracking and Re-identification: The current system processes frames independently and does not incorporate explicit player tracking or re-identification across shots. This is identified as a key area for future work to improve temporal consistency and robustness.
  • Generalizability of Player Detection and Team Assignment: While the sports field registration shows good generalizability, the player detection and team assignment modules exhibit some limitations in this regard, suggesting that fine-tuning or more robust models might be needed.
  • 3D Position Estimation: The current system focuses on 2D positional data. Extending this to 3D would provide richer analytical capabilities.
  • Adaptation to Other Sports: The authors suggest that adapting the pipeline to other sports would be an interesting future direction.

Summary updated

BibTeX
@inproceedings{theiner2022extraction,
  author = {Jonas Theiner and Wolfgang Gritz and Eric Müller-Budack and Robert Rein and Daniel Memmert and Ralph Ewerth},
  title = {{Extraction of Positional Player Data From Broadcast Soccer Videos}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure