TL;DR
This paper presents a hybrid approach for tracking individual football players in real-time video, combining the accuracy of Deep Neural Networks (DNNs) with the speed of traditional tracking methods. The proposed system achieves a good balance between accuracy and performance, outperforming existing methods by leveraging the strengths of both techniques without requiring manual intervention.
What Problem Does This Paper Address?
Tracking individual players in sports, particularly football, is a challenging task. Several factors contribute to this difficulty: players often have similar appearances, jersey numbers can be obscured, video quality can be poor due to compression, and players are frequently occluded by others. While traditional object tracking methods are fast, they lack accuracy and require manual initialization. Deep Neural Networks (DNNs) offer high accuracy but are computationally expensive, making real-time processing difficult. This paper aims to bridge this gap by developing a system that is both accurate and fast enough for real-time sports video analysis, eliminating the need for human intervention in the tracking process.
Methods
The authors propose a hybrid tracking system that combines two Convolutional Neural Network (CNN) based object detectors, Faster-RCNN and SSD, with a traditional Discriminative Correlation Filter (DCF) algorithm, specifically KCF.
- Faster-RCNN: This is a highly accurate but slower detector. It's used to process one out of every 'A' frames, acting as a "guide" for the other algorithms.
- SSD (Single Shot MultiBox Detector): This detector is less accurate than Faster-RCNN but offers better performance. It's applied to sub-frames cropped around the area where Faster-RCNN detected the player in the preceding 'A' frames.
- KCF (Kernelized Correlation Filter): This traditional algorithm is good at tracking an object over time but suffers from accuracy issues and error accumulation. In this hybrid approach, the CNNs provide continuous updates on the player's position, effectively acting as a human operator to inform KCF and correct its potential tracking errors.
This combination aims to leverage Faster-RCNN's accuracy for periodic re-localization, SSD's speed for intermediate frames, and KCF's efficiency for smooth, continuous tracking, all while mitigating the individual weaknesses of each method.
Datasets
The approach was trained and validated using the SoccerNet Dataset. This dataset consists of hand-annotated video sequences of football matches. The paper mentions using it for tracking players from two different teams.
Key Insights and Ideas
- Synergistic Combination: The core insight is that by intelligently combining the strengths of accurate but slow DNNs with fast but less accurate traditional trackers, a system can achieve both high accuracy and real-time performance.
- Automated Initialization: The hybrid system eliminates the need for a human operator to manually select the player to be tracked, a significant advantage for practical applications.
- Hierarchical Tracking Strategy: Using a powerful detector like Faster-RCNN periodically to "re-anchor" the tracking process, while relying on faster methods for intermediate frames, is an effective strategy for balancing speed and accuracy.
- DNNs as Human Operators: The paper frames the role of the DNNs in updating the KCF tracker as analogous to a human operator, highlighting how AI can automate complex tasks that previously required human intervention.
Key Results
The proposed hybrid approach was evaluated on a Nvidia GTX 1080Ti GPU.
- Performance: The system achieved an average frame rate of 87.685 FPS, with individual players being tracked at rates as high as 91.75 FPS. This demonstrates its capability for real-time processing.
- Accuracy: The average Area Under Curve (AUC) for tracking accuracy was 0.6302. This is noted as being comparable to state-of-the-art algorithms on generic datasets.
- Lost Frames: The system lost track of the target player in an average of only 0.5 frames across all tested players, indicating robust tracking.
Limitations and Future Work
- Accuracy Ceiling: While the accuracy is comparable to other methods, the paper acknowledges that the hybrid approach sacrifices some accuracy compared to using Faster-RCNN on every frame. The average AUC of 0.6302 suggests there is still room for improvement in challenging scenarios.
- Dataset Specificity: The evaluation was performed on the SoccerNet dataset. While this is a relevant benchmark, the generalizability of the approach to other sports or different video conditions might require further investigation.
- Future Work: The authors suggest that further improvements in the accuracy of the individual CNN components or exploring different combinations of detectors and trackers could lead to even better results. Optimizing the parameter 'A' (the frame interval for Faster-RCNN) could also be a direction for future research.
Summary updated