Abstract
Video-based action recognition in sports is often hindered by "semantic noise"—irrelevant objects and relationships within a scene that distract classifiers. This work proposes four novel techniques to abstract scene graphs, filtering out non-essential data to focus on action-critical components. By refining these representations, the authors achieve a 26.6% improvement in mean Average Precision (mAP) on sports datasets and demonstrate significantly faster model convergence.
Introduction
Modern computer vision models often struggle with the "signal-to-noise" ratio in complex video scenes. While Scene Graphs—graphical representations that localize objects (nodes) and their relationships (edges)—provide rich semantic information, they are often "cluttered."
In a sports context, a dense scene graph might identify every person in the stands, every chair, and every distant vehicle. For a classifier trying to identify "hitting a baseball," these extra details are distractors that add uncertainty. The challenge lies in determining which specific objects and relations a model should attend to. The authors aim to bridge the gap between raw, dense symbolic data and the high-level semantic context required for accurate action recognition.
Methods
The researchers propose a pipeline that takes raw video frames, generates a "feature bank" of scene graphs, and then applies one of four fundamentally different abstraction techniques to prune the data.
The four abstraction techniques are:
- CRF-based Abstraction: Uses Conditional Random Fields to model the statistical co-occurrence of objects and relations. It increases the weight of pairs that typically appear together (e.g., "person" + "bat" + "hitting") while suppressing unrelated combinations.
- CCA-based Abstraction: Employs Canonical Correlation Analysis to find a linear mapping between the observed scene features and the ground-truth action labels, maximizing correlation in a lower-dimensional subspace.
- Global Context-based Abstraction: This method uses natural language processing (sentence embeddings) to calculate the semantic similarity between an action label and an object-relation pair. It keeps only the top k most semantically relevant pairs.
- PCA-based Abstraction: Uses Kernel Principal Component Analysis (kPCA) for non-linear dimensionality reduction. This approach identifies the most significant components of the feature representation and discards the "least significant" ones, which are assumed to be noise.
These abstracted features are then aggregated with standard 3D CNN (pixel-level) features using an attention-based mechanism to classify the action.
Datasets and Results
The authors evaluated their methods on two primary benchmarks: Charades (daily indoor activities) and Sports Videos in the Wild (SVW) (4,200 smartphone-captured videos covering 30 sport categories).
- SVW Dataset: The proposed methods improved the mean Average Precision (mAP) for action recognition by 26.6%, reaching an mAP of 88.1% (compared to a baseline of 61.5% from prior work).
- Charades Dataset: The PCA and CCA-based abstractions performed best, achieving 72.2 mAP, outperforming the baseline "Scene-Time-Object" (STO) approach which scored 71.7.
- Qualitative Separability: Using t-SNE plots, the authors demonstrated that the abstracted features formed much more distinct, separable clusters in the feature space compared to the "raw" cluttered feature banks.
Key Insights and Conclusion
The core "aha" moment is that a smaller, more relevant knowledge representation is more powerful than a dense, exhaustive one. Removing "semantic noise" directly increases the probability of the classifier picking the correct label. Furthermore, models trained on abstracted features converge much faster, suggesting that the filtered data allows the neural network to learn underlying patterns more efficiently.
While PCA-based methods offer high performance, they sacrifice interpretability compared to Global Context methods. Future work will explore real-time inference costs and the application of these techniques to even more complex, multi-actor sports scenarios where the "clutter" problem is even more pronounced.
Summary updated