Learning Unbiased Transformer for Long-Tail Sports Action Classification

Yijun Qian, Lijun Yu, Wenhe Liu, Alexander HauptmannMediaEval 2021 · OpenReview

Introduction

Fine-grained action classification in sports videos presents a significant challenge for computer vision systems. Unlike general action recognition, sports analysis requires distinguishing between extremely subtle, high-speed movements, such as the difference between a forehand loop and a forehand flip in table tennis. This task is further complicated by two primary factors: background bias, where models overfit to static environments like the table or room, and severe data imbalance.

The Long-Tail Challenge

The MediaEval 2021 Sports Video Task dataset exemplifies these difficulties. It contains 20 distinct table tennis stroke classes, but the distribution of these classes is heavily skewed. As shown in the training set distribution, a few "head" classes dominate the data, while many "tail" classes have very few samples, and some even have zero representation in the training split. This long-tail distribution typically causes standard models to become biased toward majority classes, failing to generalize to rarer actions.

Figure 1: Distribution of stroke-wise segments in the training set, illustrating the severe long-tail nature of the data.

Methodology

To address these issues, the CMU INF team developed a three-pronged approach that secured first place in the competition:

  1. Video SWIN-Transformer (VST) Backbone: The system utilizes a Video SWIN-Transformer (VST-B) initialized with Kinetics600 weights. This architecture allows for the extraction of rich spatial-temporal features, moving beyond the limitations of traditional 3D CNNs.
  2. Background Erasing: To prevent the model from "cheating" by memorizing static background elements, the researchers implemented a background erasing strategy. By creating "distracting samples" (mixing a static frame into the clip) and applying a Mean Squared Error (MSE) loss, the model is forced to achieve feature invariance regarding the background, focusing instead on the moving human subjects.
  3. Balanced Loss: A class-wise weighted cross-entropy loss was implemented to mitigate the long-tail effect. By setting weights inversely proportional to class frequency, the model is incentivized to prioritize learning from rare stroke examples.

Results and Ablation Study

The proposed pipeline achieved a dominant 74.21% accuracy, outperforming the runner-up by over 5%. The effectiveness of each component was validated through a series of ablation runs.

Figure 3: Performance comparison across different system specifications, highlighting the impact of balanced loss and background erasing.

As indicated in the results table, the baseline VST model achieved 63.35% accuracy. The addition of a balanced loss provided a modest boost to 66.06%. However, the most significant improvement came from the background erasing technique, which resulted in an 8.15% jump in test accuracy. This underscores the critical importance of addressing environmental bias in sports video classification.

Discussion and Future Work

While the model excels at distinguishing between broad categories like forehand vs. backhand or serves vs. rallies, it still faces challenges with tactical nuances, such as differentiating offensive from defensive strokes. Furthermore, the presence of "zero-shot" classes—strokes with no training data—remains an open problem that simple loss reweighting cannot solve. Future work will focus on pretraining on more specialized fine-grained datasets and exploring sub-group attribute classification to further refine model performance.

Summary updated

BibTeX
@inproceedings{qian2021learning,
  author = {Yijun Qian and Lijun Yu and Wenhe Liu and Alexander Hauptmann},
  title = {{Learning Unbiased Transformer for Long-Tail Sports Action Classification}},
  booktitle = {MediaEval},
  year = {2021},
  url = {https://openreview.net/forum?id=jg-Ha6R4Mhl},
}
Figure