ARTHuS: Adaptive Real-Time Human Segmentation in Sports Through Online Distillation

Anthony Cioppa, Adrien Deliège, Maxime Istasse, Christophe De Vleeschouwer, Marc Van DroogenbroeckWorkshops @ CVPR 2019 · DOI

Semantic segmentation—the process of assigning a category like "human" to every pixel in an image—is a cornerstone of modern sports analytics. It enables precise player tracking, tactical analysis, and augmented reality broadcasts. However, implementing this in a live sports environment presents a "triple threat" of challenges: the lack of pixel-wise annotated data, the necessity for real-time processing, and the high variability between different matches (e.g., lighting, jersey colors, and stadium geometry).

Presented at a CVPR Workshop, ARTHuS (Adaptive Real-Time Human Segmentation in Sports) introduces a framework that overcomes these hurdles by prioritizing match-specific adaptation over universal generalization.

Methodology: Online Knowledge Distillation

The core innovation of ARTHuS is its use of online knowledge distillation. Instead of relying on a single static model, the system continuously learns the specific visual characteristics of the current match as it unfolds. This is achieved through a teacher-student architecture.

The ARTHuS framework architecture, illustrating the online distillation process between a slow teacher network and a fast adaptive student network.
  • The Teacher ($T$): A high-capacity, "universal" model (such as Mask R-CNN) that provides accurate but slow segmentation (~2 fps). It acts as an automated annotator, generating "silver standard" labels on the fly.
  • The Student ($S$): A lightweight, high-speed model (like TinyNet or ICNet) designed for real-time performance (~60 fps).
  • The Pipeline: As the video stream progresses, the Teacher processes a subset of frames to populate a first-in, first-out (FIFO) training buffer. A "training twin" of the student network ($S^{train}$) continuously learns to mimic the Teacher’s output using this sliding window of recent frames. Periodically, the optimized weights are transferred to the "inference student" ($S^{seg}$), which performs the actual real-time segmentation for the end-user.

Experimental Setup and Datasets

Due to the absence of large-scale, pixel-wise annotated sports datasets, the authors evaluated ARTHuS across two distinct professional scenarios:

  1. Soccer: A Belgian Jupiler Pro League match (FC Bruges vs. Anderlecht).
  2. Basketball: A French Jeep Elite League match (Cholet vs. Boulazac).

To validate accuracy, the authors manually corrected a subset of frames to create a "clean" ground truth. They also employed the Beucher gradient to define a "don't care" margin around human boundaries, accounting for the inherent pixel-level uncertainty in segmentation tasks.

Results and Discussion

ARTHuS demonstrated that sacrificing universality for match-specific specialization yields significant performance gains. The student network (TinyNet) achieved an inference speed of ~60 fps on an NVIDIA Tesla V100, comfortably meeting broadcast requirements.

F1 score performance over match time for soccer (top) and basketball (bottom), highlighting the system's ability to recover from unusual game events.

Key Insights

  • The Student Surpassing the Teacher: Remarkably, the student often produced cleaner segmentation masks than its teacher. By training on a continuous stream of frames, the student learns a more temporally consistent representation of "humans," effectively filtering out the frame-by-frame jitter or localized errors inherent in the Teacher's predictions.
  • Resilience to Anomalies: The system's adaptability was tested against "unusual events," such as mascots entering a basketball court. While these events initially caused a dip in performance, the online distillation process allowed the model to adapt to the new visual data and recover its accuracy within minutes.
  • Architecture Selection: While ICNet is more powerful, the smaller TinyNet proved superior for this application. Its lower parameter count allowed it to adapt much faster to changing match conditions during the online training phase.

Limitations and Future Work

While ARTHuS represents a significant step forward, several areas for improvement remain. The current FIFO buffer strategy is simple; a more "intelligent" frame selection process that retains highly informative older frames could enhance stability. Additionally, because the student is entirely dependent on the Teacher, catastrophic failures in the Teacher (e.g., due to extreme weather) would eventually propagate to the student. Future research may explore hybrid approaches that combine fixed global knowledge with this online adaptive stream to provide a more robust baseline.

Summary updated

BibTeX
@inproceedings{cioppa2019arthus,
  author = {Anthony Cioppa and Adrien Deliège and Maxime Istasse and Christophe De Vleeschouwer and Marc Van Droogenbroeck},
  title = {{ARTHuS: Adaptive Real-Time Human Segmentation in Sports Through Online Distillation}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  doi = {10.1109/cvprw.2019.00306},
}
Figure