Unifying Global and Local Scene Entities Modelling for Precise Action Spotting

Kim Hoang Tran, Phuc Vuong Do, Ngoc Quoc Ly, Ngan Le

TL;DR

This paper introduces a novel "Unifying Global and Local" (UGL) module for precise action spotting in sports videos. By disentangling global scene context from local, relevant scene entities (like balls or cards) using a Vision-Language model and an adaptive attention mechanism, the approach significantly improves accuracy, especially for challenging actions involving small objects.

What Problem Does This Paper Address?

Action spotting in sports videos is crucial for broadcasting and analysis, aiming to identify specific moments of action. However, existing methods often struggle with several challenges inherent to sports footage: cluttered backgrounds, dynamic camera angles, small objects that are key to actions (e.g., a soccer ball, a red card), and a highly imbalanced distribution of action classes. Many current approaches rely heavily on global features extracted by a "black box" backbone network, which can overlook the fine-grained details of small but critical elements within a scene. This leads to difficulties in accurately detecting actions that depend on these subtle cues.

Methods

The core innovation is the Unifying Global and Local (UGL) module. This module aims to mimic human perception by simultaneously considering the overall scene environment and specific, relevant objects within it.

  1. Global Environment Feature Extraction: A 2D CNN backbone (RegNet-Y) is used for efficiency. To capture temporal information without the computational cost of 3D CNNs, a time-shift module (GSM) is integrated. This allows for temporal feature sharing across frames.
  2. Local Relevant Entities Feature Extraction: To identify crucial scene entities, the authors leverage a Grounded Language-Image Pre-trained model (GLIP). GLIP is used to detect objects based on textual descriptions (e.g., "red card," "goal"). An Adaptive Attention Mechanism (AAM) then refines these detected entities, fusing them with the global environment features to select the most relevant ones.
  3. Fusion Component: The global environment features and local relevant entities features are concatenated and processed through a self-attention mechanism to create a unified entities-environment feature.

For temporal reasoning, a Long-term Temporal Reasoning (LTR) module is employed, using a Gated Recurrent Unit (GRU) for semantic modeling and a fully connected layer for proposal estimation. To address class imbalance, the paper adopts Focal Loss instead of Cross-Entropy.

Datasets

The paper evaluates its method on three datasets:

  • SoccerNet-v2: A large-scale dataset for action spotting in soccer videos, known for its challenges like cluttered backgrounds and small objects.
  • FineDiving: A dataset for fine-grained action understanding in diving clips.
  • FineGym: A dataset for fine-grained action understanding in gymnastics performances.

The authors note that SoccerNet-v2 exhibits significant class imbalance, with a low ratio of non-background to background frames.

Key Insights and Ideas

  • Disentangling Global and Local is Key: The most significant insight is that a holistic understanding of sports actions requires more than just global scene context. Explicitly modeling and fusing local, relevant entities (like a ball or a card) with the global environment dramatically improves performance.
  • Vision-Language Models for Entity Detection: Leveraging pre-trained Vision-Language models like GLIP is highly effective for identifying sports-specific objects without extensive task-specific training, offering robustness and generalization.
  • Adaptive Attention for Relevance: The AAM is crucial for filtering and prioritizing the most important local entities, ensuring that the model focuses on what truly matters for action recognition.
  • Efficiency Matters: Using a 2D backbone with a time-shift mechanism for temporal modeling offers a computationally efficient alternative to 3D CNNs, making the approach more practical.
  • Focal Loss for Imbalance: The adoption of Focal Loss is a practical and effective strategy for handling the severe class imbalance prevalent in sports action spotting datasets.

Key Results

The proposed method achieved state-of-the-art performance on the SoccerNet-v2 challenge set, securing 1st place.

  • It achieved 69.38% T-mAP on SoccerNet-v2, outperforming previous SOTA methods by a notable margin, including E2E-Spot by 2.65% T-mAP.
  • On FineDiving and FineGym, the method also showed significant improvements, surpassing baselines by an average of 12.03% and 15.3% mAP respectively (at Δ=2 tolerance).
  • The paper also demonstrates interpretability, allowing for an explanation of which scene elements contribute to action detection, a contrast to black-box models.

Limitations and Future Work

Limitations: * The paper acknowledges the challenge of "unshown actions" (actions not visible in the frame) where their method, like others, might struggle. * While the method shows strong performance, the ablation study on Focal Loss indicates that the hyperparameter $\gamma$ (set to 5) is crucial and might require fine-tuning for optimal results on different datasets.

Future Work: * The authors propose using Graph Neural Networks (GNNs) to model relationships between relevant scene entities, treating entities as nodes and relationships as edges. This could incorporate prior knowledge about sport rules and enhance explainability. * To further address class imbalance, they suggest grouping related classes (e.g., "yellow card," "red card," "yellow to red card" into a "cards" group) to improve classification.

Summary updated

Abstract

Sports videos pose complex challenges, including cluttered backgrounds, camera angle changes, small action-representing objects, and imbalanced action class distribution. Existing methods for detecting actions in sports videos heavily rely on global features, utilizing a backbone network as a black box that encompasses the entire spatial frame. However, these approaches tend to overlook the nuances of the scene and struggle with detecting actions that occupy a small portion of the frame. In particular, they face difficulties when dealing with action classes involving small objects, such as balls or yellow/red cards in soccer, which only occupy a fraction of the screen space. To address these challenges, we introduce a novel approach that analyzes and models scene entities using an adaptive attention mechanism. Particularly, our model disentangles the scene content into the global environment feature and local relevant scene entities feature. To efficiently extract environmental features while considering temporal information with less computational cost, we propose the use of a 2D backbone network with a time-shift mechanism. To accurately capture relevant scene entities, we employ a Vision-Language model in conjunction with the adaptive attention mechanism. Our model has demonstrated outstanding performance, securing the 1st place in the SoccerNet-v2 Action Spotting, FineDiving, and FineGym challenge with a substantial performance improvement of 1.6, 2.0, and 1.3 points in avg-mAP compared to the runner-up methods. Furthermore, our approach offers interpretability capabilities in contrast to other deep learning models, which are often designed as black boxes. Our code and models are released at: https://github.com/Fsoft-AIC/unifying-global-local-feature.
BibTeX
@inproceedings{tran2024unifying,
  author = {Kim Hoang Tran and Phuc Vuong Do and Ngoc Quoc Ly and Ngan Le},
  title = {{Unifying Global and Local Scene Entities Modelling for Precise Action Spotting}},
  booktitle = {IJCNN},
  year = {2024},
  doi = {10.1109/ijcnn60899.2024.10650009},
}
Figure