A unified and efficient framework for court-net sports video analysis using 3D camera modeling

Jungong Han, Peter H. N. de WithProceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE 2007 · DOI

TL;DR

This paper introduces a unified framework for analyzing court-net sports videos (like tennis, badminton, volleyball) by combining novel 3D camera modeling with an efficient multi-player tracking system. This allows for a more semantically rich understanding of game events, including player trajectories and heights, moving beyond simple highlight extraction.

What Problem Does This Paper Address?

The explosion of sports video content necessitates automated analysis for applications like smart retrieval and summarization. Existing sports video analysis systems often fall into one of two camps: those that focus on low-level features (like court lines or player tracking) which lack semantic understanding, or those that try to extract highlights or predefined events but struggle with generality and deep comprehension of the game's narrative. A key challenge is bridging the gap between pixel-level information and high-level semantic understanding, particularly by accurately representing the 3D spatial relationships of players and objects within the game. This paper aims to create a unified, efficient, and more generalizable framework for court-net sports that addresses these limitations.

Methods

The proposed framework operates on a multi-level analysis approach:

  • Playing-Frame Detection: Efficiently filters out non-playing frames (e.g., breaks, advertisements) by analyzing white pixel changes, focusing on actual game action.
  • Court-Net Detection and 3D Camera Modeling: This is a core contribution. Instead of just mapping 2D image planes to a standard court plane, they introduce a novel 3D camera model. This model leverages features from two perpendicular planes (the ground plane and the net plane) to establish a true 3D reference. This allows for the computation of object heights, not just their positions on the ground. The calibration process uses feature points derived from line intersections within these planes.
  • Multi-Player Tracking: A significant improvement over previous work, this algorithm addresses two key issues:
  • Template Adaptation: It uses a similarity function (Bhattacharyya coefficient) to intelligently decide when to update player templates, reducing drifting errors and computational cost compared to methods that always update or use multiple matching procedures.
  • Template Scaling: It adaptively scales player templates based on detected subregions (head and feet), which are relatively rigid and easier to find, allowing for accurate tracking even when players move towards or away from the camera.
  • Object Refinement in 3D Domain: Integrates the 3D camera model with tracking data to derive key parameters like real-world trajectories and speeds of players.
  • Scene-Level Event Classification: Uses visual cues and sport-specific knowledge to identify semantic events within the game.

Datasets

The system was evaluated on MPEG-2 video clips of badminton, tennis, and volleyball matches, totaling over 1.5 hours of regular TV broadcasts. No new datasets were introduced.

Key Insights and Ideas

  • 3D is Crucial for Semantics: Simply tracking players in 2D isn't enough. Incorporating 3D information, particularly object height derived from a robust camera model, is essential for deeper game understanding and applications like virtual view generation or ball trajectory analysis.
  • Perpendicular Planes are Key: Using both the ground and net planes as references for 3D camera calibration provides a more accurate and stable geometric transformation.
  • Smart Template Updates Save Time: For player tracking, intelligently deciding when to update a player's template (based on similarity metrics) is more efficient and robust than always updating or using complex multi-template matching.
  • Head and Feet are Reliable Anchors: Using these rigid body parts for adaptive template scaling proves effective for handling dramatic changes in player size due to camera distance.
  • Generality Through Shared Structure: The framework is designed to be adaptable to different court-net sports by changing the court-net model and specific game context knowledge, rather than requiring entirely new algorithms for each sport.

Key Results

  • High Detection Accuracy: Court-line detection achieved >92% accuracy, net-line detection >90%, and camera calibration accuracy was over 90% for most sequences.
  • Accurate Player Tracking: The multi-player tracking algorithm achieved a 95% correct tracking rate on tennis and badminton videos, even handling occlusions.
  • Efficiency Gains: The proposed template updating strategy saved approximately 15% computation time compared to a previous reference method while achieving higher quality.
  • Real-time Performance: The system demonstrated near real-time performance on a P-IV 3GHz PC, making it suitable for industrial applications.

Limitations and Future Work

  • Limitations: The paper acknowledges that detecting small objects like balls in 3D is still a difficult and ongoing research area. The current system focuses on player height and trajectory.
  • Future Work: The authors suggest incorporating a probabilistic event-classification method into the semantic-event recognition model to further improve the detection rate of semantic events. They also envision the framework's output (XML format) facilitating AV-management applications like search and retrieval.

Summary updated

Abstract

The extensive amount of video data stored on available media (hard and optical disks) necessitates video content analysis, which is a cornerstone for different user-friendly applications, such as, smart video retrieval and intelligent video summarization. This paper aims at finding a <i>unified</i> and <i>efficient</i> framework for court-net sports video analysis. We concentrate on techniques that are generally applicable for more than one sports type to come to a unified approach. To this end, our framework employs the concept of multi-level analysis, where a novel 3-D camera modeling is utilized to bridge the gap between the object-level and the scene-level analysis. The new 3-D camera modeling is based on collecting features points from two planes, which are perpendicular to each other, so that a true 3-D reference is obtained. Another important contribution is a new tracking algorithm for the objects (i.e. players). The algorithm can track up to four players simultaneously. The complete system contributes to summarization by various forms of information, of which the most important are the moving trajectory and real-speed of each player, as well as 3-D height information of objects and the semantic event segments in a game. We illustrate the performance of the proposed system by evaluating it for a variety of court-net sports videos containing badminton, tennis and volleyball, and we show that the feature detection performance is above 92% and events detection about 90%.
BibTeX
@inproceedings{han2007unified,
  author = {Jungong Han and Peter H. N. de With},
  title = {{A unified and efficient framework for court-net sports video analysis using 3D camera modeling}},
  booktitle = {Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE},
  year = {2007},
  doi = {10.1117/12.705943},
}
Figure