Efficient Golf Ball Detection and Tracking Based on Convolutional Neural Networks and Kalman Filter

Tianxiao Zhang, Xiaohan Zhang, Yiju Yang, Zongbo Wang, Guanghui WangarXiv 2020 · arXiv

Abstract

Detecting and tracking a golf ball in flight presents a significant challenge for computer vision due to the object's small size, high velocity, and the presence of motion blur. This paper presents a high-performance system that combines Convolutional Neural Networks (CNNs) with a discrete Kalman filter to achieve real-time tracking. By using the Kalman filter to predict the ball's location and cropping a small "search patch" for the CNN to analyze, the researchers achieved high accuracy while maintaining speeds up to 358 FPS.

Introduction: The "Needle in a Haystack" Problem

Detecting a golf ball in flight is a classic "needle in a haystack" problem. Golf balls are extremely small relative to high-resolution frames, often occupying as little as 10x10 pixels. Furthermore, they move at high velocities—frequently exceeding 150 mph—which results in significant motion blur.

Traditional computer vision methods, such as frame differencing or color-based tracking, often struggle in complex outdoor environments characterized by changing light conditions and cluttered backgrounds. Conversely, modern deep learning detectors face difficulties when an object occupies only a tiny fraction of the total pixels. Processing a full 4K or 1080p frame through a CNN is not only computationally expensive but often leads to the network "missing" the ball entirely due to downsampling.

Methodology: A Two-Stage Recursive Approach

To address these challenges, the authors propose a two-stage recursive framework that integrates temporal prediction with spatial detection.

The proposed tracking system combines a Kalman filter for location prediction with a CNN for refined object detection.

The system operates in four main steps:

  1. Prediction (Kalman Filter): A discrete Kalman filter predicts the ball's coordinates in the next frame based on its previous trajectory, accounting for both position and velocity.
  2. Dynamic Cropping: Instead of analyzing the entire image, the system crops a small 416 × 416 pixel patch centered on the Kalman filter’s prediction. This effectively increases the ball's resolution relative to the network's input size, allowing the CNN to "see" features that would otherwise be lost.
  3. Detection (CNN): A pre-trained CNN detector searches the small patch to refine the ball's exact location. The authors evaluated several architectures, including YOLOv3, YOLOv4, Faster R-CNN, SSD, and RefineDet.
  4. Update: The detection result is fed back into the Kalman filter to update its state for the subsequent frame.

This "Relative Size Trick" allows the system to use heavy-duty detectors like Faster R-CNN in real-time because they only process a tiny fraction of the original image data.

Datasets and Evaluation Metrics

Since no standard benchmark existed for golf balls in flight, the authors curated a custom dataset:

  • Detection Dataset: 1,699 images were collected from online tournaments and manual filming. To enhance the training set, the researchers used patch augmentation, shifting the crop around the ball to generate 11,030 training samples.
  • Tracking Dataset: 470 images across 13 video sequences (8 swings and 5 putts) were used to test temporal consistency.

The authors also highlighted an annotation challenge: at long distances, a 1-pixel labeling error can drop the Intersection over Union (IoU) by 33%. Consequently, they argued for using [email protected] as a more realistic metric for such small objects, where standard 0.50 thresholds are often physically impossible due to human labeling variance.

Key Results and Performance

The experimental results demonstrate that the proposed system is both accurate and efficient.

Visual tracking results showing the system's ability to maintain a consistent trajectory even against complex backgrounds.
  • Accuracy: YOLOv4 and Faster R-CNN were the top performers, achieving 99.3% and 98.3% [email protected] respectively.
  • Speed: YOLOv3 tiny emerged as the fastest model, reaching 357.85 FPS on an Nvidia Titan XP, making it ideal for high-frame-rate camera applications.
  • Tracking Success: Faster R-CNN achieved the highest average success rate (64.3%) in tracking evaluations, proving robust against the ball's high-speed trajectory and motion blur.

Conclusion and Future Work

The integration of a Kalman filter with CNN-based detection provides a robust solution for tracking small, fast-moving objects. While the current system assumes the ball's initial position is known, future work could involve implementing a "global search" mode for autonomous initialization. Additionally, expanding the dataset to include more diverse lighting and modeling complex ball physics (like hooks or slices) could further enhance the system's real-world utility.

Summary updated

BibTeX
@misc{zhang2020efficient,
  author = {Tianxiao Zhang and Xiaohan Zhang and Yiju Yang and Zongbo Wang and Guanghui Wang},
  title = {{Efficient Golf Ball Detection and Tracking Based on Convolutional Neural Networks and Kalman Filter}},
  year = {2020},
  eprint = {2012.09393},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.09393},
}
Figure