SBGAR: Semantics Based Group Activity Recognition

Xin Li, Mooi Choo ChuahICCV 2017 · DOI

Abstract

Recognizing group activities in video sequences is a challenging task that necessitates understanding complex interactions between multiple actors. Traditional tracking-based approaches, while effective, are often computationally expensive and sensitive to video quality. This paper introduces SBGAR (Semantics Based Group Activity Recognition), a novel two-stage framework that utilizes natural language as a mid-level semantic representation. By first generating descriptive captions for individual video frames and then classifying the overall activity based on these textual sequences, SBGAR achieves state-of-the-art accuracy on benchmark datasets while offering significant improvements in processing speed.

Introduction

While individual action recognition has seen significant progress, group activity recognition (GAR) remains difficult due to the high-level logic required to interpret collective behavior. Existing methods typically rely on tracking individual participants and modeling their spatial-temporal relationships. However, these "tracking-based" methods suffer from high computational costs and performance degradation in low-resolution or crowded scenes.

SBGAR addresses these challenges by treating activity recognition as a hierarchical language processing task. By converting visual data into semantic keywords, the model effectively filters visual noise and focuses on the core logic of the scene, bypassing the need for explicit individual tracking.

Methodology

The SBGAR framework consists of a two-stage pipeline designed to bridge the gap between raw pixels and high-level activity labels.

Stage I: Caption Generation

In the first stage, the system processes video frames to generate descriptive captions. Features are extracted using a Convolutional Neural Network (CNN)—specifically Inception-v3—from both raw RGB images and Dense Optical Flow images to capture both appearance and motion. These features are then fed into a Long Short-Term Memory (LSTM) network (LSTM1) that outputs a text caption describing the frame (e.g., "A player is jumping on the right side").

Stage II: Activity Prediction

The second stage treats the sequence of generated captions as the primary input. A sliding window of these captions is processed by a secondary CNN-LSTM architecture (CNN3 and LSTM2). This stage "reads" the sequence of semantic descriptions to predict the final group activity label, such as "Right Spike" or "Walking."

Overview of the SBGAR two-stage framework including preprocessing, caption generation, and activity prediction.

Experimental Results

The model was evaluated on two major benchmarks: the Collective Activity Dataset and the Volleyball Dataset.

Performance Metrics

SBGAR demonstrated superior performance over existing state-of-the-art methods:

  • Collective Activity Dataset: The model achieved 86.1% accuracy, surpassing the previous best of 83.4%.
  • Volleyball Dataset: The model achieved 66.9% accuracy, representing a significant improvement over the previous state-of-the-art of 51.1%.

Computational Efficiency

A key advantage of SBGAR is its speed. The system can predict a group activity label in approximately 108.5ms per 10-frame sequence. In contrast, leading tracking-based competitors require over 4 seconds for the same task on identical hardware, making SBGAR a viable candidate for real-time applications.

Extracted figure from page 6 of SBGAR: Semantics Based Group Activity Recognition

Discussion and Key Insights

  • Semantics as a Mid-level Feature: The results validate that natural language is a powerful representation for complex scenes. By using text, the model can learn spatial relationships (e.g., "Left" vs "Right" court positions) without explicit coordinate geometry.
  • Motion vs. Appearance: The authors observed that motion data (Optical Flow) is critical for fast-paced sports like volleyball, whereas raw RGB frames are more informative for slower, everyday activities like "talking" or "waiting."
  • Limitations: The model occasionally confuses sequential or visually similar activities. For instance, "waiting" is sometimes misclassified as "crossing" due to their natural temporal proximity. In volleyball, "setting" and "passing" remain difficult to distinguish due to their visual similarity.

Conclusion

SBGAR demonstrates that a semantics-based approach to group activity recognition is both highly accurate and computationally efficient. By leveraging language as a bridge, the framework avoids the pitfalls of traditional tracking methods. Future work will explore the use of automated captioning tools to reduce manual labeling requirements and the integration of more complex grammatical structures to capture nuanced human interactions.

Summary updated

BibTeX
@inproceedings{li2017sbgar,
  author = {Xin Li and Mooi Choo Chuah},
  title = {{SBGAR: Semantics Based Group Activity Recognition}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2017},
  doi = {10.1109/iccv.2017.313},
}
Figure