Abstract
Person-person mutual action recognition (also referred to as interaction recognition) is an important research branch of human activity analysis. Current solutions in the field – mainly dominated by CNNs, GCNs and LSTMs – often consist of complicated architectures and mechanisms to embed the relationships between the two persons on the architecture itself, to ensure the interaction patterns can be properly learned. Our main contribution with this work is by proposing a simpler yet very powerful architecture, named Interaction Relational Network, which utilizes minimal prior knowledge about the structure of the human body. We drive the network to identify by itself how to relate the body parts from the individuals interacting. In order to better represent the interaction, we define two different relationships, leading to specialized architectures and models for each. These multiple relationship models will then be fused into a single and special architecture, in order to leverage both streams of information for further enhancing the relational reasoning capability. Furthermore we define important structured pair-wise operations to extract meaningful extra information from each pair of joints – distance and motion. Ultimately, with the coupling of an LSTM, our IRN is capable of paramount sequential relational reasoning. Our solution is able to achieve state-of-the-art performance on the traditional interaction recognition datasets SBU and UT, and also on the mutual actions from the large-scale dataset NTU RGB+D.
Introduction
Recognizing how two people interact—whether they are punching, hugging, or shaking hands—is a complex computer vision task essential for human activity analysis. Most current solutions rely on "heavy" architectures like Graph Convolutional Networks (GCNs) or complex LSTMs that are hard-coded with prior knowledge of the human skeleton. While powerful, these models often struggle to explicitly capture the nuanced relationships between the body parts of different people. For example, a "kick" is defined not just by the movement of one person's leg, but by the spatial relationship between that leg and the other person's body.
The Interaction Relational Network (IRN) addresses this by moving away from rigid, hand-crafted body structures. Instead, it learns to relate body parts across individuals dynamically, allowing the network to identify which joint pairings matter most for a specific action.
Methods
The IRN is built on the concept of Relational Networks (RNs), treating every skeletal joint as an independent "object" and learning a function that describes how pairs of these objects relate.
Key innovations in the IRN architecture include:
- Dual Relationship Modeling: The network splits reasoning into two specialized streams: Intra-person (how a person’s own joints move relative to each other) and Inter-person (how one person’s joints move relative to the other person’s). These are modeled by sub-networks ($g_\theta$) and then fused into a single architecture ($f_{\phi''}$).
- Structured Pair-wise Input: Rather than using raw coordinates alone, the input is augmented by explicitly calculating the distance and motion between every pair of joints. This provides the network with vital domain-specific primitives without requiring a complex, rigid architecture.
- Temporal Reasoning: To handle the sequential nature of video, the IRN is coupled with an LSTM, allowing the model to reason about how these joint relationships evolve over time.
Datasets and Evaluation
The model was evaluated on four major benchmarks:
- SBU Interaction Dataset: A Kinect-based dataset featuring 8 types of two-person interactions.
- UT-Interaction: An RGB-based dataset where joint coordinates were estimated using OpenPose.
- NTU RGB+D: A large-scale dataset focusing on 11 mutual action classes.
- NTU RGB+D 120: An extension featuring 26 mutual action classes and more complex environmental conditions.
Key Results and Insights
The IRN achieved impressive quantitative results, often setting new state-of-the-art (SOTA) benchmarks. It reached 98.2% accuracy on SBU and 98.3% on UT-Interaction. On the large-scale NTU RGB+D, it obtained 90.5% (Cross-Subject), outperforming many graph-based methods.
A critical insight from the research was the power of fusion. A "naive" approach of mixing all joints together performed poorly. The real breakthrough came from realizing that Inter and Intra relationships carry distinct semantic meanings. As shown in the confusion matrices above, combining these streams significantly reduces misclassifications for actions like "Pushing" or "Exchanging," which rely heavily on both individual movement and interpersonal coordination.
Limitations and Future Work
While highly effective, the IRN occasionally confuses actions that appear similar in skeleton-only space but involve different objects, such as "Hit with object" versus "Punching."
Future research directions include:
- Multimodal Integration: Incorporating RGB visual features to help identify objects involved in interactions.
- Attention Mechanisms: Implementing attention to help the model focus on the most informative joint pairs (e.g., hands during a handshake).
- Group Activity Recognition: Adapting the IRN framework to model relationships between three or more people.
Summary updated