Introduction
Fine-grained action classification in sports videos presents a significant challenge for computer vision systems. Unlike general action recognition, sports analysis requires distinguishing between extremely subtle, high-speed movements, such as the difference between a forehand loop and a forehand flip in table tennis. This task is further complicated by two primary factors: background bias, where models overfit to static environments like the table or room, and severe data imbalance.
The Long-Tail Challenge
The MediaEval 2021 Sports Video Task dataset exemplifies these difficulties. It contains 20 distinct table tennis stroke classes, but the distribution of these classes is heavily skewed. As shown in the training set distribution, a few "head" classes dominate the data, while many "tail" classes have very few samples, and some even have zero representation in the training split. This long-tail distribution typically causes standard models to become biased toward majority classes, failing to generalize to rarer actions.
Methodology
To address these issues, the CMU INF team developed a three-pronged approach that secured first place in the competition:
- Video SWIN-Transformer (VST) Backbone: The system utilizes a Video SWIN-Transformer (VST-B) initialized with Kinetics600 weights. This architecture allows for the extraction of rich spatial-temporal features, moving beyond the limitations of traditional 3D CNNs.
- Background Erasing: To prevent the model from "cheating" by memorizing static background elements, the researchers implemented a background erasing strategy. By creating "distracting samples" (mixing a static frame into the clip) and applying a Mean Squared Error (MSE) loss, the model is forced to achieve feature invariance regarding the background, focusing instead on the moving human subjects.
- Balanced Loss: A class-wise weighted cross-entropy loss was implemented to mitigate the long-tail effect. By setting weights inversely proportional to class frequency, the model is incentivized to prioritize learning from rare stroke examples.
Results and Ablation Study
The proposed pipeline achieved a dominant 74.21% accuracy, outperforming the runner-up by over 5%. The effectiveness of each component was validated through a series of ablation runs.
As indicated in the results table, the baseline VST model achieved 63.35% accuracy. The addition of a balanced loss provided a modest boost to 66.06%. However, the most significant improvement came from the background erasing technique, which resulted in an 8.15% jump in test accuracy. This underscores the critical importance of addressing environmental bias in sports video classification.
Discussion and Future Work
While the model excels at distinguishing between broad categories like forehand vs. backhand or serves vs. rallies, it still faces challenges with tactical nuances, such as differentiating offensive from defensive strokes. Furthermore, the presence of "zero-shot" classes—strokes with no training data—remains an open problem that simple loss reweighting cannot solve. Future work will focus on pretraining on more specialized fine-grained datasets and exploring sub-group attribute classification to further refine model performance.
Summary updated