Hierarchical Graph-Rnns for Action Detection of Multiple Activities

Sovan Biswas, Yaser Souri, Juergen Gall

TL;DR

This paper introduces a novel hierarchical Graph-RNN approach for detecting multiple simultaneous actions performed by individuals in videos. By jointly modeling temporal scene context and inter-person action relationships, the method achieves state-of-the-art performance on the challenging AVA dataset.

What Problem Does This Paper Address?

Action detection in videos, especially when multiple actions occur simultaneously and by different individuals, is a complex task. Existing methods often struggle with:

  • Spatio-temporal localization: Accurately identifying where and when actions happen.
  • Multiple simultaneous actions: A single person can perform several actions at once (e.g., "standing" and "talking on the phone").
  • Inter-person interactions: Actions are not always independent; they can involve interactions with others (e.g., "talking to" another person).
  • Data annotation cost: Fully annotating every frame with bounding boxes and action labels is prohibitively expensive. Datasets like AVA use temporally sparse annotations (one frame per second), requiring models that can learn from this limited supervision.

This paper aims to address these challenges by developing a model that can effectively leverage both temporal information within a scene and the relational dynamics between detected individuals, even with sparse annotations.

Methods

The proposed approach, a Hierarchical Graph-RNN (HGRNN), consists of two main components trained jointly:

  1. Scene RNN (Lower Level): This component models the temporal context of the entire video frame. It takes I3D features (a standard for video understanding) from the current and neighboring frames as input and outputs a hidden state representing the scene's temporal context. This is implemented using a bidirectional GRU.
  2. Graph RNN (Upper Level): This component models the relationships between detected persons. Each detected person is treated as a node in a fully connected graph. The Graph RNN iteratively updates the hidden representation of each person's node by considering their own features, the scene context from the Scene RNN, and the intermediate representations of all other persons in the graph. This allows it to learn how actions of different individuals influence each other.

The model does not rely on tracked bounding boxes, which can be unreliable, but instead uses detected bounding boxes and their associated features. The final output for each detected person is a set of action labels, inferred by a fully connected layer with a sigmoid activation.

Datasets

  • AVA 2.1 dataset: This is the primary dataset used for training and evaluation. It's notable for its large scale and temporally sparse annotations, where actions and bounding boxes are labeled for only one frame per second. This dataset contains 60 action classes.

Key Insights and Ideas

  • Joint modeling is crucial: The paper demonstrates that combining temporal scene context (via Scene RNN) and inter-person action relationships (via Graph RNN) leads to significantly better performance than using either component alone.
  • Interactions matter: Actions are not isolated events. Understanding how people interact with each other is key to accurately detecting their activities, especially in complex group scenarios.
  • Leveraging sparse annotations: The hierarchical approach is designed to effectively learn from temporally sparse annotations, a common challenge in large-scale video datasets.
  • Temporal context aids recognition: Longer temporal receptive fields help in recognizing actions that might be subtle or unfold over time (e.g., "open" and "close").

Key Results

  • State-of-the-art performance: The proposed HGRNN achieves state-of-the-art results on the AVA 2.1 dataset, outperforming previous methods by a significant margin (e.g., 6.4% on RGB data compared to a vanilla I3D head).
  • Impact of temporal context: Increasing the temporal context length (l) from 1 to 3 seconds (corresponding to 7 seconds of video) improved accuracy, highlighting the importance of temporal information.
  • Efficiency of Graph RNN: The Graph RNN requires only a few iterations (specifically, 2 iterations were found to be sufficient) to effectively update node representations, suggesting an efficient learning process for interactions.
  • Detector accuracy impact: Using ground truth bounding boxes instead of detected ones improved action detection accuracy by 5-6%, indicating that the performance of the person detector is a significant factor.
  • RGB vs. RGB+Flow: Incorporating optical flow features further boosted accuracy, but the proposed RGB-only method already achieved competitive results.

Limitations and Future Work

  • Reliance on person detection: The performance is inherently tied to the accuracy of the initial person detector. Errors in detection can propagate to action detection.
  • Future direction: Integrated detector: The authors suggest that integrating the action detection directly into a single Faster RCNN framework, rather than using a separate detector and then the HGRNN, could be a promising avenue for future research.
  • Optical flow: While the paper shows gains with optical flow, it primarily focuses on RGB data. Exploring the full potential of RGB+Flow within their HGRNN framework is a natural next step.

Summary updated

Abstract

In this paper, we propose an approach that spatially lo-calizes the activities in a video frame where each person can perform multiple activities at the same time. Our approach takes the temporal scene context as well as the relations of the actions of detected persons into account. While the temporal context is modeled by a temporal recurrent neural network (RNN), the relations of the actions are modeled by a graph RNN. Both networks are trained together and the proposed approach achieves state of the art results on the AVA dataset.
BibTeX
@inproceedings{biswas2019hierarchical,
  author = {Sovan Biswas and Yaser Souri and Juergen Gall},
  title = {{Hierarchical Graph-Rnns for Action Detection of Multiple Activities}},
  booktitle = {ICIP},
  year = {2019},
  doi = {10.1109/icip.2019.8803650},
}
Figure