Learning Mutual Excitation for Hand-to-Hand and Human-to-Human Interaction Recognition

Mengyuan Liu, Chen Chen, Songtao Wu, Fanyang Meng, Hong Liu

Abstract

Recognizing interactive actions, including hand-to-hand interaction and human-to-human interaction, has attracted increasing attention for various applications in the field of video analysis and human-robot interaction. Considering the success of graph convolution in modeling topology-aware features from skeleton data, recent methods commonly operate graph convolution on separate entities and use late fusion for interactive action recognition, which can barely model the mutual semantic relationships between pairwise entities. To this end, we propose a mutual excitation graph convolutional network (me-GCN) by stacking mutual excitation graph convolution (me-GC) layers. Specifically, me-GC uses a mutual topology excitation module to firstly extract adjacency matrices from individual entities and then adaptively model the mutual constraints between them. Moreover, me-GC extends the above idea and further uses a mutual feature excitation module to extract and merge deep features from pairwise entities. Compared with graph convolution, our proposed me-GC gradually learns mutual information in each layer and each stage of graph convolution operations. Extensive experiments on a challenging hand-to-hand interaction dataset, i.e., the Assembely101 dataset, and two large-scale human-to-human interaction datasets, i.e., NTU60-Interaction and NTU120-Interaction consistently verify the superiority of our proposed method, which outperforms the state-of-the-art GCN-based and Transformer-based methods.

1. Introduction

Skeleton-based action recognition has seen significant progress through Graph Convolutional Networks (GCNs), which represent the human body as a natural graph of joints and bones. However, recognizing interactive actions—such as a handshake, a hug, or complex assembly tasks—remains a distinct challenge. Most existing GCN-based approaches follow a "split-and-fusion" pipeline: they process the skeletons of interacting entities (e.g., two hands or two people) independently through separate layers and only combine their information at the final stage (late fusion).

Conceptual comparison between traditional split-and-fusion GCN pipelines and the proposed mutual excitation approach for interactive actions.

As illustrated in Figure 1, this late-fusion approach often overlooks the mutual semantic relationships between body parts. For example, the distinction between a "high-five" and a "handshake" lies primarily in the relative motion and spatial constraints between the two hands. By delaying interaction modeling until the end of the network, standard models fail to capture these fine-grained, interdependent motion patterns.

2. The Mutual Excitation Graph Convolutional Network (me-GCN)

To address these limitations, we propose the Mutual Excitation Graph Convolutional Network (me-GCN). The core innovation is the mutual excitation graph convolution (me-GC) layer, which replaces standard graph convolutions to learn mutual information throughout every stage of the network.

The me-GCN architecture overview, featuring the Mutual Topology Excitation (MTE) and Mutual Feature Excitation (MFE) modules within the me-GC layer.

2.1 Mutual Topology Excitation (MTE)

Instead of using a fixed or independently learned graph structure for each entity, the MTE module (Figure 3d) adaptively models the constraints between them. It calculates both intra-correlation (relationships within one person's joints) and inter-correlation (relationships between joints of different people). A learnable parameter, $\beta$, dynamically fuses these correlations to create a "mutual" topology that highlights the joints most relevant to the interaction.

2.2 Mutual Feature Excitation (MFE)

While MTE focuses on the structure of the graph, the MFE module (Figure 3e) focuses on the joint features. It extracts deep features from both entities, computes a shared context through feature averaging, and then injects this mutual information back into the individual feature paths. This acts as a cross-entity attention mechanism, "exciting" relevant joints while suppressing noise from irrelevant body parts.

3. Experimental Evaluation

The me-GCN was evaluated on three large-scale datasets: Assemble101 (hand-to-hand interaction), NTU60-Interaction, and NTU120-Interaction (human-to-human interaction).

3.1 Key Results

The proposed model consistently outperformed state-of-the-art GCN and Transformer-based methods:

  • Assemble101: Achieved 27.2% accuracy, a significant 4.7% lead over the previous best model (IGFormer).
  • NTU120-Interaction: Reached 90.0% accuracy on both Cross-Subject and Cross-Setup protocols, outperforming the Graph Transformer-based 2sKA-AGTN by up to 3.3%.
  • NTU60-Interaction: Achieved near-saturation results with 98.2% (Cross-View) and 95.5% (Cross-Subject) accuracy.

4. Discussion and Conclusion

Our analysis reveals that mutual learning serves as a powerful attention mechanism. By considering both participants simultaneously, the model learns to focus on "critical joints"—for example, increasing activation scores for hands during a handshake while suppressing irrelevant joints like feet.

Furthermore, ablation studies demonstrated that the "cascade" approach of me-GCN—sharing information at every layer—is superior to both early and late fusion. Early fusion can destroy individual motion patterns, while late fusion misses low-level mutual cues. The me-GCN finds the optimal balance by maintaining separate feature paths that are continuously refined by mutual information. While currently designed for pairwise interactions, future work will explore extending this architecture to multi-person scenarios.

Summary updated

Abstract

Recognizing interactive actions, including hand-to-hand interaction and human-to-human interaction, has attracted increasing attention for various applications in the field of video analysis and human-robot interaction. Considering the success of graph convolution in modeling topology-aware features from skeleton data, recent methods commonly operate graph convolution on separate entities and use late fusion for interactive action recognition, which can barely model the mutual semantic relationships between pairwise entities. To this end, we propose a mutual excitation graph convolutional network (me-GCN) by stacking mutual excitation graph convolution (me-GC) layers. Specifically, me-GC uses a mutual topology excitation module to firstly extract adjacency matrices from individual entities and then adaptively model the mutual constraints between them. Moreover, me-GC extends the above idea and further uses a mutual feature excitation module to extract and merge deep features from pairwise entities. Compared with graph convolution, our proposed me-GC gradually learns mutual information in each layer and each stage of graph convolution operations. Extensive experiments on a challenging hand-to-hand interaction dataset, i.e., the Assembely101 dataset, and two large-scale human-to-human interaction datasets, i.e., NTU60-Interaction and NTU120-Interaction consistently verify the superiority of our proposed method, which outperforms the state-of-the-art GCN-based and Transformer-based methods.
BibTeX
@article{liu2025learning,
  author = {Mengyuan Liu and Chen Chen and Songtao Wu and Fanyang Meng and Hong Liu},
  title = {{Learning Mutual Excitation for Hand-to-Hand and Human-to-Human Interaction Recognition}},
  journal = {IEEE Transactions on Human-Machine Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=ahypqrpIk2},
}
Figure