HAtt-Flow: Hierarchical Attention-Flow Mechanism for Group-Activity Scene Graph Generation in Videos

Naga Venkata Sai Raviteja Chappa, Pha A. Nguyen, Thi Hoang Ngan Le, Page Daniel Dobbs, Khoa Luu

TL;DR

This paper introduces HAtt-Flow, a novel hierarchical attention-flow mechanism for generating scene graphs of group activities in videos. By treating attention as a flow network, it improves the prediction of complex relationships between people and objects, outperforming existing methods on a newly proposed dataset with richer annotations.

What Problem Does This Paper Address?

Understanding complex visual scenes, especially in videos, is a significant challenge. Specifically, generating scene graphs that describe the relationships between objects and subjects over time (Video Scene Graph Generation or VidSGG) is crucial for tasks like activity recognition. Traditional VidSGG methods often focus on retrospective analysis, limiting their ability to predict future interactions. Furthermore, existing datasets often lack the nuanced annotations needed to capture the intricacies of group activities, where multiple individuals interact with each other and their environment. This paper aims to address these limitations by developing a more predictive VidSGG approach and introducing a richer dataset for group activity scene graph generation.

Methods

The core of the proposed method is the HAtt-Flow (Hierarchical Attention-Flow) mechanism. It's built upon three key modules:

  • Input Preparation: This module combines both textual and visual features to create node and edge embeddings for a graph transformer layer. This allows for a more holistic representation of the scene.
  • Hierarchical Awareness Induction: This module enhances attention mechanisms by incorporating a "hierarchy-aware" approach. It recursively merges visual patches and text tokens based on semantic and spatial similarities, creating increasingly concentrated clusters. This ensures that information aggregation is preserved and not lost as it moves up the hierarchy.
  • Feature Flow-Attention Mechanism: Inspired by flow network theory, this is the most novel aspect. Instead of traditional attention where "queries" attend to "keys" and "values," HAtt-Flow treats "values" as "sources" and "keys" as "sinks." This flow-based approach uses principles of flow conservation to create a competitive mechanism for sources and an allocation mechanism for sinks. This helps prevent the generation of "trivial attention" (i.e., attention that doesn't contribute meaningfully) and encourages more robust feature selection.

The authors also introduce a specific training loss formulation to encourage joint learning of textual and visual features.

Datasets

The paper introduces a new dataset called GASG (Group-Activity Scene Graph). This dataset is an extension of the JRDB dataset and features:

  • Nuanced Annotations: It includes five key aspects: appearance, situation, position, interaction, and relationships.
  • Group Activity Focus: It specifically captures individual actions, sub-group activities, and overall group activities, with a focus on interpersonal dynamics and collective behaviors.
  • Richness: It contains 27 categories of individual actions, 11 sub-group activities, and 7 group activities.

The GASG dataset was used for training and evaluation, and comparisons were also made against the PSG dataset.

Key Insights and Ideas

  • Attention as Flow: The most significant insight is reframing attention as a flow network. This allows for a more principled way to manage information flow and competition, leading to more meaningful feature selection.
  • Hierarchy is Key: Explicitly incorporating hierarchical awareness into the attention mechanism helps the model better understand multi-level dependencies and semantic structures within group activities.
  • Data Richness Matters: The introduction of the GASG dataset with its detailed annotations highlights the need for more comprehensive data to tackle the complexities of group activity scene graph generation.
  • Text Guides Vision: Ablation studies suggest that allowing attention to flow from text to vision (T→X) yields better results, implying that textual context can effectively guide the interpretation of visual information in group activities.

Key Results

The HAtt-Flow model demonstrated state-of-the-art performance on the newly introduced GASG dataset.

  • GASG Dataset Performance: The paper reports strong results on the GASG dataset, outperforming existing methods across various metrics (R/mR@20, R/mR@50, R/mR@100) for both predicate classification (PredCls) and video scene graph generation (VSGG). For example, on the GASG dataset, the proposed method achieved 57.4/35.2 for PredCls and 42.2/21.4 for VSGG at R/mR@20.
  • PSG Dataset Performance: The model also showed competitive results on the PSG dataset, achieving state-of-the-art performance on most metrics.
  • Qualitative Improvements: Visualizations show that HAtt-Flow generates more accurate and complete scene graphs, correctly identifying subjects, groups, and their interactions, unlike previous methods that might only detect subjects.

Limitations and Future Work

  • Computational Demands: The flow-attention mechanism, while effective, imposes significant computational demands, which could be a challenge for deployment in resource-constrained environments. Future work could focus on optimizing these algorithms and exploring hardware acceleration.
  • Data Dependency: The model's performance is sensitive to the quality and quantity of training data. Variations in data quality or insufficient samples can impact robustness. Future research should explore methods to enhance robustness to data variations.
  • Flow Direction: While T→X flow direction was found to be optimal, the exact reasons for performance decline in other directions warrant further investigation.
  • Attribute Interplay: Further research into the interplay between different attributes and their impact on performance could lead to model architecture refinements.

Summary updated

Abstract

Group Activity Scene Graph (GASG) generation is a challenging task in computer vision, aiming to anticipate and describe relationships between subjects and objects in video sequences. Traditional Video Scene Graph Generation (VidSGG) methods focus on retrospective analysis, limiting their predictive capabilities. To enrich the scene understanding capabilities, we introduced a GASG dataset extending the JRDB dataset with nuanced annotations involving \textit{Appearance, Interaction, Position, Relationship, and Situation} attributes. This work also introduces an innovative approach, \textbf{H}ierarchical \textbf{Att}ention-\textbf{Flow} (HAtt-Flow) Mechanism, rooted in flow network theory to enhance GASG performance. Flow-Attention incorporates flow conservation principles, fostering competition for sources and allocation for sinks, effectively preventing the generation of trivial attention. Our proposed approach offers a unique perspective on attention mechanisms, where conventional "values" and "keys" are transformed into sources and sinks, respectively, creating a novel framework for attention-based models. Through extensive experiments, we demonstrate the effectiveness of our Hatt-Flow model and the superiority of our proposed Flow-Attention mechanism. This work represents a significant advancement in predictive video scene understanding, providing valuable insights and techniques for applications that require real-time relationship prediction in video data.
BibTeX
@misc{chappa2023hatt,
  author = {Naga Venkata Sai Raviteja Chappa and Pha A. Nguyen and Thi Hoang Ngan Le and Page Daniel Dobbs and Khoa Luu},
  title = {{HAtt-Flow: Hierarchical Attention-Flow Mechanism for Group-Activity Scene Graph Generation in Videos}},
  year = {2023},
  eprint = {2312.07740},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.07740},
}
Figure