Semantic segmentation—the process of assigning a category like "human" to every pixel in an image—is a cornerstone of modern sports analytics. It enables precise player tracking, tactical analysis, and augmented reality broadcasts. However, implementing this in a live sports environment presents a "triple threat" of challenges: the lack of pixel-wise annotated data, the necessity for real-time processing, and the high variability between different matches (e.g., lighting, jersey colors, and stadium geometry).
Presented at a CVPR Workshop, ARTHuS (Adaptive Real-Time Human Segmentation in Sports) introduces a framework that overcomes these hurdles by prioritizing match-specific adaptation over universal generalization.
Methodology: Online Knowledge Distillation
The core innovation of ARTHuS is its use of online knowledge distillation. Instead of relying on a single static model, the system continuously learns the specific visual characteristics of the current match as it unfolds. This is achieved through a teacher-student architecture.
- The Teacher ($T$): A high-capacity, "universal" model (such as Mask R-CNN) that provides accurate but slow segmentation (~2 fps). It acts as an automated annotator, generating "silver standard" labels on the fly.
- The Student ($S$): A lightweight, high-speed model (like TinyNet or ICNet) designed for real-time performance (~60 fps).
- The Pipeline: As the video stream progresses, the Teacher processes a subset of frames to populate a first-in, first-out (FIFO) training buffer. A "training twin" of the student network ($S^{train}$) continuously learns to mimic the Teacher’s output using this sliding window of recent frames. Periodically, the optimized weights are transferred to the "inference student" ($S^{seg}$), which performs the actual real-time segmentation for the end-user.
Experimental Setup and Datasets
Due to the absence of large-scale, pixel-wise annotated sports datasets, the authors evaluated ARTHuS across two distinct professional scenarios:
- Soccer: A Belgian Jupiler Pro League match (FC Bruges vs. Anderlecht).
- Basketball: A French Jeep Elite League match (Cholet vs. Boulazac).
To validate accuracy, the authors manually corrected a subset of frames to create a "clean" ground truth. They also employed the Beucher gradient to define a "don't care" margin around human boundaries, accounting for the inherent pixel-level uncertainty in segmentation tasks.
Results and Discussion
ARTHuS demonstrated that sacrificing universality for match-specific specialization yields significant performance gains. The student network (TinyNet) achieved an inference speed of ~60 fps on an NVIDIA Tesla V100, comfortably meeting broadcast requirements.
Key Insights
- The Student Surpassing the Teacher: Remarkably, the student often produced cleaner segmentation masks than its teacher. By training on a continuous stream of frames, the student learns a more temporally consistent representation of "humans," effectively filtering out the frame-by-frame jitter or localized errors inherent in the Teacher's predictions.
- Resilience to Anomalies: The system's adaptability was tested against "unusual events," such as mascots entering a basketball court. While these events initially caused a dip in performance, the online distillation process allowed the model to adapt to the new visual data and recover its accuracy within minutes.
- Architecture Selection: While ICNet is more powerful, the smaller TinyNet proved superior for this application. Its lower parameter count allowed it to adapt much faster to changing match conditions during the online training phase.
Limitations and Future Work
While ARTHuS represents a significant step forward, several areas for improvement remain. The current FIFO buffer strategy is simple; a more "intelligent" frame selection process that retains highly informative older frames could enhance stability. Additionally, because the student is entirely dependent on the Teacher, catastrophic failures in the Teacher (e.g., due to extreme weather) would eventually propagate to the student. Future research may explore hybrid approaches that combine fixed global knowledge with this online adaptive stream to provide a more robust baseline.
Summary updated