Multimodal and multiview distillation for real-time player detection on a football field

Anthony Cioppa, Adrien Deliège, Noor ul Huda, Rikke Gade, Marc Van Droogenbroeck, Thomas B. MoeslundWorkshops @ CVPR 2020 · DOI

This paper, originally presented at a CVPR 2020 workshop, offers a clever solution to a common computer vision headache: how to monitor a massive area like a football field without breaking the bank on hardware or drowning in manual labeling.

Abstract

Monitoring large-scale environments typically requires expensive multi-camera setups or labor-intensive manual labeling. This paper presents a novel "teacher-student" distillation framework that leverages a single high-cost thermal camera to train a low-cost, wide-angle fisheye camera for real-time player detection. By integrating mathematical distortion modeling for data augmentation and motion detection to manage unlabeled regions, the system effectively learns to monitor the entire field, despite the teacher's limited field of view.

The Challenge: Distortion and Cost

Monitoring a football field typically requires a complex, expensive array of standard cameras. While a single fisheye camera can capture the entire field, it introduces two major hurdles: extreme lens distortion (players look different depending on where they stand) and low resolution for distant subjects.

Thermal cameras are excellent for person detection because they are invariant to lighting changes, but they are expensive and usually have a narrow field of view (FOV). The authors proposed a Knowledge Distillation approach to solve this: using high-quality detections from a thermal camera to automatically train a fisheye camera to monitor the whole field.

Methodology

In this setup, a "Teacher" network (YOLOv3) processes thermal images, and a "Student" network (YOLOv3-tiny) processes fisheye RGB images. The core innovation lies in how the researchers handled the FOV mismatch. The thermal camera only covered about 22% of the field (the "overlapping region").

The multimodal setup: a thermal camera with a narrow FOV acts as a teacher for a wide-angle fisheye camera.

To train the student to detect players in the remaining 78% (the "non-overlapping region"), they used two key techniques:

1. Geometry-Aware Data Augmentation

They took player "crops" from the overlapping region and pasted them into the non-overlapping region. Crucially, they used polar coordinates to mathematically rescale and rotate the crops to mimic the specific lens distortion of the fisheye camera at that new location. This turns a hardware limitation (distortion) into a predictable mathematical transform.

The data augmentation pipeline: (a) extraction from the overlap, (b) scaling, (c) rotation based on lens geometry, and (d) blending into the full fisheye view.

2. Motion-Aware Loss

A major problem with training on the non-overlapping region is that real players are already there, but they don't have ground-truth labels from the teacher. To prevent the student from being penalized for correctly detecting these "unlabeled" players, the authors used a motion detection algorithm (ViBe). If the student detected something in a moving area, the loss was zeroed out, effectively telling the model: "We aren't sure if a player is there, so don't learn anything from this specific spot right now."

Experimental Results

The study used a custom dataset recorded at an amateur football field at night with artificial lighting. To validate the system, the authors manually annotated the ground truth for one frame every 10 seconds.

  • Counting Accuracy: The system achieved a Root Mean Square Error (RMSE) of approximately 3.4 players. Given that a football match has 22 players plus officials, this is a highly usable estimate for occupancy monitoring.
  • Online Adaptation: The system performs "online distillation," meaning the student is constantly being updated by the teacher during the game. This allows the model to adapt to changing weather or lighting conditions on the fly.
  • Ablation Success: The "Full Method" (Augmentation + Motion Masking) significantly outperformed simpler versions. Without the motion mask, the student was overwhelmed by false positives; without the augmentation, it couldn't detect players at the edges of the field.

Limitations and Future Work

While promising, the evaluation was performed on a single one-hour video, and it remains to be seen how well this generalizes to different field types. Additionally, the authors used a low Intersection over Union (IoU) threshold of 0.25, reflecting the difficulty of precisely bounding very small, distorted players. Future work may also need to explicitly address player occlusions.

Overall, this paper provides a robust blueprint for "frugal" AI—using a small amount of expensive data to unlock the potential of cheap, ubiquitous hardware.

Summary updated

BibTeX
@inproceedings{cioppa2020multimodal,
  author = {Anthony Cioppa and Adrien Deliège and Noor ul Huda and Rikke Gade and Marc Van Droogenbroeck and Thomas B. Moeslund},
  title = {{Multimodal and multiview distillation for real-time player detection on a football field}},
  booktitle = {Workshops @ CVPR},
  year = {2020},
  doi = {10.1109/cvprw50498.2020.00448},
}
Figure