Consistency-Aware Graph Network for Human Interaction Understanding

Zhenhua Wang, Jiajun Meng, Dongyan Guo, Jianhua Zhang, Qinfeng (Javen) Shi, Shengyong Chen

Abstract

Human Interaction Understanding (HIU) requires more than simple action recognition; it necessitates a coherent understanding of the relationships between multiple actors. Traditional models often produce logically inconsistent results, such as predicting conflicting actions between the same pair of individuals. This paper introduces LAGNet (Logic-Aware Graph Network), a framework that enforces logical consistency through a third-order graph network and a differentiable reasoning module. By integrating "logic oracles" directly into the neural architecture, LAGNet ensures that predictions obey fundamental rules of compatibility and group transitivity, achieving state-of-the-art results across major benchmarks.

Introduction

Understanding human interaction is a cornerstone of social artificial intelligence. It involves identifying complex semantic links, such as recognizing that if Person A is kicking Person B, then Person B must be being kicked. Current deep learning models frequently struggle with this, treating individual actions and pairwise interactions as independent tasks. This independence often leads to "hallucinated" inconsistencies where the predicted labels for a group of people do not form a logically sound scene.

Conventional models often produce inconsistent predictions (right) compared to the ground truth (middle), such as failing to recognize that a person not involved in an interaction (C) should be labeled as 'no-interaction'.

Methodology

The proposed LAGNet architecture consists of three integrated stages designed to move beyond simple pairwise analysis:

  1. Base Feature Extraction: A standard CNN (Inception V3) extracts visual features from regions of interest (RoIs) corresponding to detected individuals.
  2. Third-Order Graph Network (TOGN): Unlike standard graph networks that focus on pairs, TOGN models dependencies among triplets. This allows the network to capture richer contextual information regarding how multiple people relate within a social group.
  3. Consistency-Aware Reasoning (CAR) Module: This module acts as a differentiable "logic engine." It utilizes an energy function embedded with two specific oracles:
  4. Compatibility Oracle: Ensures action labels are semantically paired (e.g., matching "kick" with "be-kicked").
  5. Transitivity Oracle: Enforces group logic, ensuring that if A interacts with B and B interacts with C, the system recognizes them as part of the same interacting cluster.
The LAGNet architecture integrates a Third-Order Graph Network with a differentiable reasoning module to ensure logical consistency in interaction predictions.

Experimental Results

LAGNet was evaluated on three standard HIU benchmarks: UT Interaction, BIT-Interaction, and TV Human Interaction (TVHI). The model achieved state-of-the-art (SOTA) performance across the board:

  • TVHI Dataset: LAGNet reached an F1-score of 92.83%, a significant improvement over the previous SOTA of approximately 83.5%.
  • BIT Dataset: The model achieved a 92.79% F1-score, outperforming other recent graph-based approaches.
  • Ablation Studies: Removing the CAR module resulted in a performance drop of up to 4.6 points on TVHI, proving that explicit logical enforcement significantly aids the model's visual perception.

Conclusion

LAGNet demonstrates that logical constraints can be effectively integrated into deep learning architectures as differentiable layers rather than mere post-processing steps. By framing common-sense reasoning as an energy minimization problem, the model successfully resolves the logical gaps prevalent in traditional HIU systems. Future research will focus on simultaneous end-to-end training of the backbone and the reasoning modules, as well as incorporating temporal dynamics to better capture how interactions evolve over time.

Summary updated

Abstract

Compared with the progress made on human activity classification, much less success has been achieved on human interaction understanding (HIU). Apart from the latter task is much more challenging, the main cause is that recent approaches learn human interactive relations via shallow graphical models, which is inadequate to model complicated human interactions. In this paper, we propose a consistency-aware graph network, which combines the representative ability of graph network and the consistency-aware reasoning to facilitate the HIU task. Our network consists of three components, a backbone CNN to extract image features, a factor graph network to learn third-order interactive relations among participants, and a consistency-aware reasoning module to enforce labeling and grouping consistencies. Our key observation is that the consistency-aware-reasoning bias for HIU can be embedded into an energy function, minimizing which delivers consistent predictions. An efficient mean-field inference algorithm is proposed, such that all modules of our network could be trained jointly in an end-to-end manner. Experimental results show that our approach achieves leading performance on three benchmarks.
BibTeX
@inproceedings{wang2021consistency,
  author = {Zhenhua Wang and Jiajun Meng and Dongyan Guo and Jianhua Zhang and Qinfeng (Javen) Shi and Shengyong Chen},
  title = {{Consistency-Aware Graph Network for Human Interaction Understanding}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01312},
}
Figure