Body-Hand Modality Expertized Networks with Cross-attention for Fine-grained Skeleton Action Recognition

Seungyeon Cho, Tae-Kyun Kim

TL;DR

This paper introduces BHaRNet, a novel dual-stream network that excels at fine-grained skeleton-based action recognition by treating body and hand movements as separate "expert" modalities. By using cross-attention for feature interaction and an ensemble loss, BHaRNet achieves state-of-the-art accuracy while being more computationally efficient than existing unified approaches, and it also extends effectively to multi-modal (skeleton + RGB) recognition.

What Problem Does This Paper Address?

Human Action Recognition (HAR) using skeleton data is crucial for applications like robotics and human-robot interaction. While current methods often use a unified graph representation of the entire body, they tend to overlook the subtle nuances of hand movements, which are critical for distinguishing fine-grained actions (e.g., "Okay sign" vs. "Victory sign"). This is because the pooling operations used to handle the complexity of unified graphs can blur these fine hand details, especially when contrasted with larger, more dominant body movements. The paper aims to bridge this gap by developing a model that can effectively capture both global body dynamics and intricate hand articulations.

Methods

The core of BHaRNet is a dual-stream architecture that processes body and hand skeleton data independently.

  • Body-Expert and Hand-Expert Streams: Each stream uses a Graph Convolutional Network (GCN) backbone to learn features specific to its modality. This separation is key to preserving the integrity of fine-grained hand details.
  • Ensemble Loss: The model is jointly trained using individual classification losses for each stream and a complementary loss on the fused prediction. This encourages cooperative specialization, akin to a Mixture-of-Experts (MoE), where each expert (body/hand) contributes its specialized knowledge.
  • Cross-Attention for Feature Interaction: To enable interaction between the body and hand streams, the paper proposes two mechanisms:
  • Pooling Attention Module: This module performs structured pooling along different axes of the feature maps before applying a fast attention mechanism. This balances computational efficiency with rich spatio-temporal information exchange.
  • Expertized Branch Model: This approach further refines the dual-stream idea by splitting each stream into an "Expertized" branch (preserving modality-specific features) and an "Interactive" branch (which incorporates cross-attention). This allows for explicit specialization while still enabling controlled feature exchange.
  • Multi-Modal Extension: Inspired by MMNet, BHaRNet can integrate RGB data. Body-expert features are used to guide an RGB stream, helping it focus on relevant spatio-temporal regions and enhancing overall recognition.

Datasets

The paper evaluates BHaRNet on the following large-scale skeleton-based action recognition benchmarks:

  • NTU RGB+D 60
  • NTU RGB+D 120
  • PKU-MMD
  • Northwestern-UCLA

For multi-modal experiments, it uses NTU60, NTU120, and PKU-MMD. The paper also notes the use of Mediapipe for extracting 2.5D hand skeletons from RGB frames due to the absence of hand skeleton data in some datasets.

Key Insights and Ideas

  • Separation is Key for Fine-Grained Details: The most significant insight is that treating body and hand as separate "expert" modalities, rather than merging them into a single graph early on, is crucial for preserving the subtle details needed for fine-grained action recognition.
  • Mixture-of-Experts Synergy: The ensemble loss and cooperative training effectively create a "mixture of experts" where specialized knowledge from body and hand streams is leveraged without one overpowering the other.
  • Controlled Feature Exchange: Cross-attention, particularly with the proposed Pooling Attention Module and Expertized Branches, offers a sophisticated way to allow modalities to inform each other without losing their individual strengths.
  • Body Features as RGB Guidance: Using body skeleton features to guide RGB processing is an effective way to leverage complementary information, showing the power of multi-modal fusion when guided by strong skeleton representations.

Key Results

  • State-of-the-Art Accuracy: BHaRNet achieves state-of-the-art (SOTA) accuracies on large-scale benchmarks, notably improving performance on hand-intensive actions from 86.4% to 93.0%.
  • Efficiency Gains: The proposed models (BHaRNet-P, BHaRNet-E, BHaRNet-M) maintain fewer GFLOPs and parameters compared to relevant unified methods, demonstrating a strong efficiency-accuracy trade-off. For example, on NTU 120 X-Sub, BHaRNet-P achieves 94.3% accuracy with nearly half the GFLOPs of BlockGCN.
  • Improved Hand-Specific Recognition: Confusion matrix analysis shows BHaRNet-E exhibits notably fewer misclassifications on hand-specific actions like "OK sign" and "Victory sign" compared to SkeleT.
  • Competitive Multi-Modal Performance: In multi-modal settings, BHaRNet-M consistently achieves strong results and remains more parameter- and FLOP-efficient than typical CNN-based multi-modal pipelines.

Limitations and Future Work

  • Performance on Smaller Datasets: The paper acknowledges that BHaRNet's accuracy on the smaller N-UCLA benchmark is lower than some specialized models. This is attributed to the dataset's limited size and scarcity of hand-centric classes, which can reduce the benefits of the hand expert stream.
  • Future Directions: The authors suggest future work could focus on improving the reliability of the hand stream, especially in low-data scenarios. They also propose exploring hybrid pooling strategies, incorporating additional modalities (like depth or inertial sensors), and further refining the expertized architecture for even more accurate and lightweight multi-modal action recognition.

Summary updated

Abstract

Skeleton-based Human Action Recognition (HAR) is a vital technology in robotics and human-robot interaction. However, most existing methods concentrate primarily on full-body movements and often overlook subtle hand motions that are critical for distinguishing fine-grained actions. Recent work leverages a unified graph representation that combines body, hand, and foot keypoints to capture detailed body dynamics. Yet, these models often blur fine hand details due to the disparity between body and hand action characteristics and the loss of subtle features during the spatial-pooling. In this paper, we propose BHaRNet (Body-Hand action Recognition Network), a novel framework that augments a typical body-expert model with a hand-expert model. Our model jointly trains both streams with an ensemble loss that fosters cooperative specialization, functioning in a manner reminiscent of a Mixture-of-Experts (MoE). Moreover, cross-attention is employed via an expertized branch method and a pooling-attention module to enable feature-level interactions and selectively fuse complementary information. Inspired by MMNet, we also demonstrate the applicability of our approach to multi-modal tasks by leveraging RGB information, where body features guide RGB learning to capture richer contextual cues. Experiments on large-scale benchmarks (NTU RGB+D 60, NTU RGB+D 120, PKU-MMD, and Northwestern-UCLA) demonstrate that BHaRNet achieves SOTA accuracies -- improving from 86.4\% to 93.0\% in hand-intensive actions -- while maintaining fewer GFLOPs and parameters than the relevant unified methods.
BibTeX
@misc{cho2025body,
  author = {Seungyeon Cho and Tae-Kyun Kim},
  title = {{Body-Hand Modality Expertized Networks with Cross-attention for Fine-grained Skeleton Action Recognition}},
  year = {2025},
  eprint = {2503.14960},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.14960},
}
Figure