Detecting Events and Key Actors in Multi-person Videos

Vignesh Ramanathan, Jonathan Huang, Sami Abu-El-Haija, Alexander N. Gorban, Kevin Murphy, Li Fei-Fei

TL;DR

This paper introduces a novel attention-based model for recognizing events in multi-person videos, which can also identify the "key actors" responsible for the event without explicit supervision. They also release a new, large-scale basketball dataset with dense temporal event annotations to address the limitations of existing benchmarks.

What Problem Does This Paper Address?

Recognizing events in videos is a well-studied problem, but most existing work focuses on single-person actions. Real-world videos, like those from sports or public spaces, often feature many people, with only a subset actively contributing to the main event. Identifying these key individuals and understanding their role is crucial for accurate event recognition. Current methods struggle with this, and existing datasets are often too small or lack the detailed annotations needed for multi-person event analysis.

Methods

The core of the proposed method is an attention mechanism that learns to focus on the most relevant individuals in a video frame by frame. The model first extracts features for each detected person and then processes these features using a Bidirectional Long Short-Term Memory (BLSTM) network to capture temporal information for each individual track. An attention mechanism then dynamically weighs these individual features to create a contextually relevant representation for the entire scene. This attended representation is fed into another RNN for event classification.

The authors also propose two variants of their attention model: one that uses person tracking to maintain consistent representations across frames, and a "tracking-free" version that treats each frame's detections independently. This tracking-free approach is particularly interesting as it can adapt more readily to changing attention focus.

Datasets

The authors introduce a new dataset: the NCAA Basketball Dataset. This dataset comprises 257 basketball games, totaling approximately 1.5 hours of video per game. It features dense temporal annotations for 11 distinct event classes, with over 14,000 event annotations in total. Crucially, it also includes annotations for ball location and player bounding boxes, making it suitable for both event recognition and player localization tasks. This dataset is presented as being comparable in annotation density to existing benchmarks like THUMOS, but with significantly longer videos.

Key Insights and Ideas

  • Attention as a "Weakly Supervised" Key Actor Identifier: The model learns to identify important individuals without being explicitly told who they are. This is a significant step towards more interpretable and efficient video understanding models.
  • Dynamic Attention is Crucial: The attention weights change over time, allowing the model to shift focus from, for example, the shooter at the start of a shot to other players involved in the play as the event unfolds.
  • Tracking-Free Attention Offers Flexibility: The tracking-free attention model shows strong performance and suggests that robust event recognition can be achieved even without the complexities of maintaining precise object tracks, which can be prone to errors in crowded or fast-moving scenes.
  • Basketball as a Rich Domain for Multi-Person Events: The choice of basketball highlights the complexity of multi-person interactions and provides a good testbed for developing and evaluating models for this challenging scenario.

Key Results

The proposed attention-based model, particularly the version with tracking, outperforms state-of-the-art methods on their newly created basketball dataset for both event classification and event detection. For example, in event classification on isolated clips, their "Our track" model achieved a mean average precision (mAP) of 0.516, outperforming methods like IDT and LRCN. In event detection on untrimmed videos, their "Attn track" model achieved an mAP of 0.435, again surpassing other baselines.

Qualitatively, the attention mechanism was shown to consistently focus on relevant players, such as the shooter during a shot attempt. The tracking-free attention model was particularly effective at identifying the "shooter" in various event types, achieving a mean average precision of 0.618 for this specific task.

Limitations and Future Work

The authors acknowledge that performance can vary significantly across event classes, especially for those with very few annotated examples (e.g., "slam dunk fail"). They also note that the tracking-based attention model can sometimes be "reluctant to switch attention," potentially biasing it towards players who are consistently visible throughout the video.

Future work could involve extending these techniques to other multi-person domains and exploring more sophisticated ways to handle occlusions and complex interactions. The release of their new dataset is also a significant contribution that will enable further research in this area.

Summary updated

Abstract

Multi-person event recognition is a challenging task, often with many people active in the scene but only a small subset contributing to an actual event. In this paper, we propose a model which learns to detect events in such videos while automatically "attending" to the people responsible for the event. Our model does not use explicit annotations regarding who or where those people are during training and testing. In particular, we track people in videos and use a recurrent neural network (RNN) to represent the track features. We learn time-varying attention weights to combine these features at each time-instant. The attended features are then processed using another RNN for event detection/ classification. Since most video datasets with multiple people are restricted to a small number of videos, we also collected a new basketball dataset comprising 257 basketball games with 14K event annotations corresponding to 11 event classes. Our model outperforms state-of-the-art methods for both event classification and detection on this new dataset. Additionally, we show that the attention mechanism is able to consistently localize the relevant players.
BibTeX
@inproceedings{ramanathan2016detecting,
  author = {Vignesh Ramanathan and Jonathan Huang and Sami Abu-El-Haija and Alexander N. Gorban and Kevin Murphy and Li Fei-Fei},
  title = {{Detecting Events and Key Actors in Multi-person Videos}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2016},
  doi = {10.1109/cvpr.2016.332},
}
Figure