Abstract
Human Interaction Understanding (HIU) requires more than simple action recognition; it necessitates a coherent understanding of the relationships between multiple actors. Traditional models often produce logically inconsistent results, such as predicting conflicting actions between the same pair of individuals. This paper introduces LAGNet (Logic-Aware Graph Network), a framework that enforces logical consistency through a third-order graph network and a differentiable reasoning module. By integrating "logic oracles" directly into the neural architecture, LAGNet ensures that predictions obey fundamental rules of compatibility and group transitivity, achieving state-of-the-art results across major benchmarks.
Introduction
Understanding human interaction is a cornerstone of social artificial intelligence. It involves identifying complex semantic links, such as recognizing that if Person A is kicking Person B, then Person B must be being kicked. Current deep learning models frequently struggle with this, treating individual actions and pairwise interactions as independent tasks. This independence often leads to "hallucinated" inconsistencies where the predicted labels for a group of people do not form a logically sound scene.
Methodology
The proposed LAGNet architecture consists of three integrated stages designed to move beyond simple pairwise analysis:
- Base Feature Extraction: A standard CNN (Inception V3) extracts visual features from regions of interest (RoIs) corresponding to detected individuals.
- Third-Order Graph Network (TOGN): Unlike standard graph networks that focus on pairs, TOGN models dependencies among triplets. This allows the network to capture richer contextual information regarding how multiple people relate within a social group.
- Consistency-Aware Reasoning (CAR) Module: This module acts as a differentiable "logic engine." It utilizes an energy function embedded with two specific oracles:
- Compatibility Oracle: Ensures action labels are semantically paired (e.g., matching "kick" with "be-kicked").
- Transitivity Oracle: Enforces group logic, ensuring that if A interacts with B and B interacts with C, the system recognizes them as part of the same interacting cluster.
Experimental Results
LAGNet was evaluated on three standard HIU benchmarks: UT Interaction, BIT-Interaction, and TV Human Interaction (TVHI). The model achieved state-of-the-art (SOTA) performance across the board:
- TVHI Dataset: LAGNet reached an F1-score of 92.83%, a significant improvement over the previous SOTA of approximately 83.5%.
- BIT Dataset: The model achieved a 92.79% F1-score, outperforming other recent graph-based approaches.
- Ablation Studies: Removing the CAR module resulted in a performance drop of up to 4.6 points on TVHI, proving that explicit logical enforcement significantly aids the model's visual perception.
Conclusion
LAGNet demonstrates that logical constraints can be effectively integrated into deep learning architectures as differentiable layers rather than mere post-processing steps. By framing common-sense reasoning as an energy minimization problem, the model successfully resolves the logical gaps prevalent in traditional HIU systems. Future research will focus on simultaneous end-to-end training of the backbone and the reasoning modules, as well as incorporating temporal dynamics to better capture how interactions evolve over time.
Summary updated