TL;DR
This paper introduces a novel attention-based model for recognizing events in multi-person videos, which can also identify the "key actors" responsible for the event without explicit supervision. They also release a new, large-scale basketball dataset with dense temporal event annotations to address the limitations of existing benchmarks.
What Problem Does This Paper Address?
Recognizing events in videos is a well-studied problem, but most existing work focuses on single-person actions. Real-world videos, like those from sports or public spaces, often feature many people, with only a subset actively contributing to the main event. Identifying these key individuals and understanding their role is crucial for accurate event recognition. Current methods struggle with this, and existing datasets are often too small or lack the detailed annotations needed for multi-person event analysis.
Methods
The core of the proposed method is an attention mechanism that learns to focus on the most relevant individuals in a video frame by frame. The model first extracts features for each detected person and then processes these features using a Bidirectional Long Short-Term Memory (BLSTM) network to capture temporal information for each individual track. An attention mechanism then dynamically weighs these individual features to create a contextually relevant representation for the entire scene. This attended representation is fed into another RNN for event classification.
The authors also propose two variants of their attention model: one that uses person tracking to maintain consistent representations across frames, and a "tracking-free" version that treats each frame's detections independently. This tracking-free approach is particularly interesting as it can adapt more readily to changing attention focus.
Datasets
The authors introduce a new dataset: the NCAA Basketball Dataset. This dataset comprises 257 basketball games, totaling approximately 1.5 hours of video per game. It features dense temporal annotations for 11 distinct event classes, with over 14,000 event annotations in total. Crucially, it also includes annotations for ball location and player bounding boxes, making it suitable for both event recognition and player localization tasks. This dataset is presented as being comparable in annotation density to existing benchmarks like THUMOS, but with significantly longer videos.
Key Insights and Ideas
- Attention as a "Weakly Supervised" Key Actor Identifier: The model learns to identify important individuals without being explicitly told who they are. This is a significant step towards more interpretable and efficient video understanding models.
- Dynamic Attention is Crucial: The attention weights change over time, allowing the model to shift focus from, for example, the shooter at the start of a shot to other players involved in the play as the event unfolds.
- Tracking-Free Attention Offers Flexibility: The tracking-free attention model shows strong performance and suggests that robust event recognition can be achieved even without the complexities of maintaining precise object tracks, which can be prone to errors in crowded or fast-moving scenes.
- Basketball as a Rich Domain for Multi-Person Events: The choice of basketball highlights the complexity of multi-person interactions and provides a good testbed for developing and evaluating models for this challenging scenario.
Key Results
The proposed attention-based model, particularly the version with tracking, outperforms state-of-the-art methods on their newly created basketball dataset for both event classification and event detection. For example, in event classification on isolated clips, their "Our track" model achieved a mean average precision (mAP) of 0.516, outperforming methods like IDT and LRCN. In event detection on untrimmed videos, their "Attn track" model achieved an mAP of 0.435, again surpassing other baselines.
Qualitatively, the attention mechanism was shown to consistently focus on relevant players, such as the shooter during a shot attempt. The tracking-free attention model was particularly effective at identifying the "shooter" in various event types, achieving a mean average precision of 0.618 for this specific task.
Limitations and Future Work
The authors acknowledge that performance can vary significantly across event classes, especially for those with very few annotated examples (e.g., "slam dunk fail"). They also note that the tracking-based attention model can sometimes be "reluctant to switch attention," potentially biasing it towards players who are consistently visible throughout the video.
Future work could involve extending these techniques to other multi-person domains and exploring more sophisticated ways to handle occlusions and complex interactions. The release of their new dataset is also a significant contribution that will enable further research in this area.
Summary updated