HAtt-Flow: Hierarchical Attention-Flow Mechanism for Group Activity Scene Graph Generation in Videos

Naga Venkata Sai Raviteja Chappa, Pha Nguyen, T. Hoang Ngan Le, Khoa Luu

Introduction

Understanding complex group activities in video—such as a crowd interacting in a public space—poses a significant challenge for computer vision. Traditional Video Scene Graph Generation (VidSGG) often focuses on isolated object pairs, failing to capture the broader context of group dynamics. Moreover, many models suffer from "trivial attention," where importance is distributed uniformly across a scene, obscuring nuanced relationships.

To address these issues, researchers have developed HAtt-Flow, a framework that treats attention as a regulated fluid in a network. By enforcing "flow conservation," HAtt-Flow moves beyond simple retrospective analysis toward a predictive, hierarchical understanding of group activities.

Methodology: Hierarchical Awareness and Feature Flow

The core of HAtt-Flow lies in its ability to model scenes at multiple levels of abstraction. Unlike previous methods that treat all entities equally, HAtt-Flow employs two primary innovations:

  1. Hierarchical Awareness Induction: The model recursively merges video patches and text tokens into higher-level clusters based on spatial and semantic similarity. This bottom-up approach ensures that the model understands a "person" as a coherent entity before attempting to classify their relationships within a group.
  2. Feature Flow Attention: Inspired by flow network theory, the authors redefine the standard Attention mechanism. By treating "values" as sources and "keys" as sinks, they apply flow conservation principles. This forces a "competition" for attention, preventing the generation of blurred, non-specific attention maps and ensuring that the most relevant connections are prioritized.
Comparison of HAtt-Flow's hierarchical scene graph generation (Levels 0-3) against traditional flat relationship modeling.

The architecture utilizes a dual-branch setup (Vision and Language) optimized via a contrastive CLIP-style loss to align video features with descriptive linguistic concepts.

The Group Activity Scene Graph (GASG) Dataset

A major contribution of this work is the introduction of the Group Activity Scene Graph (GASG) dataset. Built as an extension of the JRDB dataset, GASG provides a level of detail previously unavailable in the field.

The GASG dataset provides multi-layered annotations across appearance, interaction, position, relationship, and situation.

The dataset includes 54 video sequences annotated with five distinct attribute layers: Appearance, Interaction, Position, Relationship, and Situation. This multi-layered approach allows the model to distinguish between simple proximity and complex social interactions like "talking to" or "riding along with."

Key Results and Insights

HAtt-Flow significantly outperforms existing state-of-the-art (SOTA) methods like MOTIFS and VCTree. On the GASG dataset's "Predicate Classification" (PredCls) task, HAtt-Flow achieved an R@20 of 57.4, a substantial improvement over the previous benchmark of approximately 45.

Key insights from the study include:

  • Attention as a Finite Resource: Framing attention as a flow network effectively solves the "trivial attention" problem. By treating information as a fluid that cannot be created or destroyed, the model is forced to identify the most critical social and spatial links.
  • Linguistic Guidance: Ablation studies showed that information flow from the text branch to the vision branch is crucial. High-level concepts (like "conversing") provide the necessary context to resolve visual ambiguities in dense crowds.
  • Structural Integrity: The "non-splittable" property of the hierarchy ensures that merged tokens remain grouped in deeper layers, preserving structural information throughout the processing pipeline.

Conclusion

While HAtt-Flow sets a new standard for group activity understanding, it faces challenges regarding computational demand due to the complexity of flow-attention calculations. Future research will likely focus on optimizing these calculations for real-time performance and testing the model's robustness against "long-tail" or rare social interactions in larger datasets. By bridging the gap between low-level detection and high-level social reasoning, HAtt-Flow represents a significant step forward in automated scene understanding.

Summary updated

Abstract

Group Activity Scene Graph (GASG) generation is a challenging task in computer vision, aiming to anticipate and describe relationships between subjects and objects in video sequences. Traditional Video Scene Graph Generation (VidSGG) methods focus on retrospective analysis, limiting their predictive capabilities. To enrich the scene understanding capabilities, we introduced a GASG dataset extending the JRDB dataset with nuanced annotations involving \textit{Appearance, Interaction, Position, Relationship, and Situation} attributes. This work also introduces an innovative approach, \textbf{H}ierarchical \textbf{Att}ention-\textbf{Flow} (HAtt-Flow) Mechanism, rooted in flow network theory to enhance GASG performance. Flow-Attention incorporates flow conservation principles, fostering competition for sources and allocation for sinks, effectively preventing the generation of trivial attention. Our proposed approach offers a unique perspective on attention mechanisms, where conventional "values" and "keys" are transformed into sources and sinks, respectively, creating a novel framework for attention-based models. Through extensive experiments, we demonstrate the effectiveness of our Hatt-Flow model and the superiority of our proposed Flow-Attention mechanism. This work represents a significant advancement in predictive video scene understanding, providing valuable insights and techniques for applications that require real-time relationship prediction in video data.
BibTeX
@misc{chappa2023hatt,
  author = {Naga Venkata Sai Raviteja Chappa and Pha Nguyen and T. Hoang Ngan Le and Khoa Luu},
  title = {{HAtt-Flow: Hierarchical Attention-Flow Mechanism for Group Activity Scene Graph Generation in Videos}},
  year = {2023},
  eprint = {2312.07740},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.07740},
}
Figure