Multi-Perspective Representation to Part-Based Graph for Group Activity Recognition

Lifang Wu, Xianglong Lang, Ye Xiang, Qi Wang, Meng TianSensors 2022 · OpenReview

TL;DR

This paper tackles the challenge of group activity recognition by proposing a novel part-based graph representation that captures fine-grained interactions between individual body parts. By modeling both intra-actor (within an individual) and inter-actor (between individuals) relationships, their method achieves state-of-the-art results on benchmark datasets, highlighting the importance of detailed body part analysis over holistic features.

What Problem Does This Paper Address?

Recognizing group activities, like a team playing volleyball or a group queuing, is significantly harder than identifying individual actions. While existing methods focus on individual actions and their overall relationships, they often overlook the crucial role of specific body parts (e.g., hands, legs) and their interactions. These fine-grained cues are often more discriminative for understanding complex group behaviors, especially when individuals within a group might have similar appearances or perform similar basic actions. The paper aims to bridge this gap by developing a method that leverages these detailed part-level interactions for more accurate group activity recognition.

Methods

The core of this work is a novel part-based graph representation. They propose a two-branch framework to capture both static (single frame) and dynamic (video sequence) representations.

  • Part Feature Extraction: Instead of using holistic features for each person, they extract feature maps for individual body parts.
  • Intra-Actor Part Graph: For each individual, they build a graph to model the spatial relationships between their own body parts. This helps in understanding the structural information of an individual's pose.
  • Inter-Actor Part Graph: This module models relationships between individuals. It considers two types of relations:
  • Visual Relation: Uses a mutual attention mechanism to capture appearance similarity between corresponding parts of different actors.
  • Location Relation: Leverages the relative spatial distance between actors.
  • Graph Convolutional Networks (GCNs): Both intra- and inter-actor graphs utilize GCNs to propagate information and refine features based on these part-level relationships.
  • Two-Branch Framework: A static branch (using 2D CNNs like Inception-v3) and a dynamic branch (using 3D CNNs like I3D on RGB and optical flow) are employed. The outputs from these branches are fused late in the process.

Datasets

The proposed method is evaluated on two widely used datasets:

  • Volleyball Dataset (VD): Contains clips from volleyball games with group activity and individual action labels.
  • Collective Activity Dataset (CAD): Features video sequences of various group activities like crossing, walking, queuing, talking, and waiting.

Key Insights and Ideas

  • Parts over Wholes: The paper strongly suggests that focusing on individual body parts and their interactions provides more discriminative information for group activity recognition than relying solely on holistic features of individuals.
  • Intra-Actor Structure Matters: Modeling the internal structure of an individual's pose (how their parts relate to each other) is crucial for robust individual representations, which in turn benefits group activity recognition.
  • Multi-Perspective Inter-Actor Relations: Combining visual similarity and spatial proximity between corresponding parts of different actors offers a richer understanding of group dynamics.
  • Static and Dynamic Fusion: Effectively combining static appearance information with dynamic temporal information leads to improved performance.

Key Results

  • Volleyball Dataset: Achieved a classification accuracy of 94.8%, outperforming previous state-of-the-art methods.
  • Collective Activity Dataset: Improved accuracy by 0.3% compared to existing state-of-the-art results.
  • Ablation studies demonstrated that removing either the intra-actor or inter-actor part graph modules led to performance degradation, confirming the effectiveness of their proposed components.
  • The number of parts (k x k) also impacts performance, with k=3 (9 parts) showing optimal results.

Limitations and Future Work

  • Dataset Specificity: While performing well on benchmark datasets, the generalizability to highly diverse or unconstrained real-world scenarios might still be a challenge.
  • Computational Cost: The use of multiple branches and graph convolutions can be computationally intensive.
  • Future Work: The authors suggest exploring how to combine fine-grained information with semantic information more effectively and extending the approach to other scenarios like pedestrian search and vehicle/pedestrian re-identification.

Summary updated

Abstract

Group activity recognition that infers the activity of a group of people is a challenging task and has received a great deal of interest in recent years. Different from individual action recognition, group activity recognition needs to model not only the visual cues of individuals but also the relationships between them. The existing approaches inferred relations based on the holistic features of the individual. However, parts of the human body, such as the head, hands, legs, and their relationships, are the critical cues in most group activities. In this paper, we establish the part-based graphs from different viewpoints. The intra-actor part graph is designed to model the spatial relations of different parts for an individual, and the inter-actor part graph is proposed to explore part-level relations among actors, in which visual relation and location relation are both considered. Furthermore, a two-branch framework is utilized to capture the static spatial and dynamic temporal representations simultaneously. On the Volleyball Dataset, our approach obtains a classification accuracy of 94.8%, achieving very competitive performance in comparison with the state of the art. As for the Collective Activity Dataset, our approach improves the accuracy by 0.3% compared with the state-of-the-art results.
BibTeX
@inproceedings{wu2022multi,
  author = {Lifang Wu and Xianglong Lang and Ye Xiang and Qi Wang and Meng Tian},
  title = {{Multi-Perspective Representation to Part-Based Graph for Group Activity Recognition}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=NbYfmyRR8x},
}
Figure