Abstract
Indoor action recognition remains a significant challenge due to the inherent limitations of individual data modalities. Skeleton-based models capture motion dynamics but lack object context, while RGB-based models often overfit to static backgrounds. This paper presents the Teacher-Student Multimodal Fusion (TSMF) network, which leverages skeleton-derived spatial attention to guide an RGB-based student network toward informative regions of interest. By focusing on interactions between the subject and objects, TSMF achieves state-of-the-art performance on the NTU RGB+D and PKU-MMD datasets, particularly in distinguishing fine-grained actions like "typing" versus "writing."
Introduction
Recognizing human actions in indoor environments is a critical task for ambient intelligence and robotics. However, current systems often struggle to distinguish between actions with similar motion patterns. For instance, "reading a book" and "writing on paper" involve nearly identical joint movements, making them indistinguishable to skeleton-only models. Conversely, standard RGB video models frequently overfit to static indoor backgrounds, failing to isolate the fine-grained movements of the person from the environment. The TSMF framework addresses these issues by using the structural precision of skeletons to direct the video model's attention toward discriminative appearance features.
Methodology
The TSMF architecture employs a dual-modality approach consisting of a Teacher and a Student network:
- The Teacher (Skeleton Network): Utilizing a Spatio-Temporal Graph Convolutional Network (ST-GCN), the Teacher processes 3D joint coordinates to extract spatial attention weights. These weights identify which joints (e.g., hands, head, feet) are most relevant to the current action.
- The Student (RGB Network): A ResNet-based architecture receives a fused representation of the video. Instead of processing full frames, the system uses the Teacher's attention weights to crop Regions of Interest (ROIs) around the head, hands, and feet.
- Fusion Mechanism: By concatenating these cropped ROIs into a single feature map, the model forces the Student network to ignore irrelevant background noise and focus on the interaction between the subject and objects. The final prediction is an ensemble of both networks' outputs.
Key Insights and Complementary Strengths
The primary innovation of TSMF is the use of skeleton data as a "denoiser" for RGB video. While previous methods focused primarily on hand movements, TSMF incorporates the head and feet, enabling better recognition of complex actions like "wearing a shoe."
As shown in the figure above, the RGB student compensates for the skeleton teacher's "object blindness." When the skeleton model cannot distinguish between "reading" and "writing" due to similar poses, the RGB component provides the necessary visual evidence of the object (e.g., a book or a pen) to make an accurate classification.
Experimental Results
The TSMF model was evaluated on two large-scale benchmarks: NTU RGB+D and PKU-MMD.
- NTU RGB+D: The model achieved an average accuracy of 95.0%, representing a 7.3% improvement over previous multimodal methods.
- PKU-MMD: TSMF reached 96.8% accuracy, a 3.4% improvement over the previous state-of-the-art.
- Fine-Grained Accuracy: For notoriously difficult actions like "typing" and "writing," the TSMF ensemble improved accuracy by 25-26% compared to using a skeleton-only model (ST-GCN).
Conclusion and Future Work
The Teacher-Student Multimodal Fusion network demonstrates that skeleton-guided attention significantly enhances the robustness of RGB-based action recognition by filtering out background noise and focusing on object interactions. While currently optimized for indoor settings and dependent on high-quality skeleton extraction, future research will explore extending this framework to dynamic outdoor environments and more complex datasets like Kinetics.
Summary updated