Abstract
Multimodal action recognition systems, which fuse RGB video and skeletal joint data, typically achieve superior performance on clean datasets. However, these systems exhibit significant vulnerability to adversarial attacks, where perturbations in the RGB stream can disproportionately degrade the overall classification accuracy. This paper introduces the Attention-based Modality Reweighter (AMR), a plug-and-play module designed to enhance adversarial robustness. By employing a dynamic attention mechanism, AMR learns to prioritize the more robust skeletal modality over the vulnerable RGB stream during inference. Experimental evaluations on the NTU-RGB+D 60 and iMiGUE datasets demonstrate that AMR significantly outperforms existing defense methods, achieving up to a 43.77% improvement in accuracy under PGD attacks while maintaining competitive performance on clean data.
Introduction
Deep neural networks have revolutionized action recognition, but their susceptibility to adversarial attacks remains a critical barrier to deployment in security-sensitive applications. Adversarial attacks involve human-imperceptible perturbations added to input data that cause models to fail. While single-modality defense is well-studied, multimodal systems present unique challenges. In RGB-skeleton fusion, the system often suffers from a "weakest link" problem: if the RGB modality is easily fooled, its high-dimensional noise can derail the entire prediction, even if the skeletal stream remains accurate.
This research addresses the challenge of making multimodal systems robust without sacrificing clean-data accuracy. The authors identify that skeletal data—a high-level abstraction of body joints—is inherently more resistant to the texture and background-based perturbations that plague raw RGB video. The proposed Attention-based Modality Reweighter (AMR) leverages this insight to dynamically shield the model from adversarial noise.
Proposed Method: Attention-based Modality Reweighter (AMR)
The core contribution is the AMR module, a modular component that can be integrated into standard two-stream architectures. Unlike static fusion methods, AMR uses an attention mechanism to evaluate the reliability of latent features from both the RGB and skeletal branches.
Key Technical Components
- Dynamic Weighting: AMR assigns importance weights to features. When the system detects (through learned patterns) that a modality is noisy or unreliable, it suppresses those features and amplifies the more stable ones—typically the skeletal features.
- Deep Supervision: To ensure robustness is learned throughout the network, the module generates auxiliary predictions at intermediate layers. These are incorporated into a novel loss function, forcing the model to develop robust representations at multiple levels of abstraction rather than just at the final output layer.
- Plug-and-Play Architecture: The design is highly flexible. In the study, multiple AMR modules were inserted into standard backbones like I3D (for RGB) and HCN (for skeleton), demonstrating its compatibility with existing state-of-the-art models.
Experimental Results
The authors evaluated AMR against various adversarial attacks, including Fast Gradient Sign Method (FGSM), Projected Gradient Descent (PGD), and Carlini & Wagner (CW) attacks, across two major benchmarks: NTU-RGB+D 60 and the micro-gesture dataset iMiGUE.
The Robustness Gap
Empirical analysis confirmed the "weakest link" hypothesis. As shown in the performance curves below, the accuracy of the RGB stream (black) and standard multimodal fusion (green) drops precipitously as attack intensity ($\\epsilon$) increases. In contrast, the skeletal modality (blue) maintains significantly higher accuracy.
Performance Gains
On the NTU-RGB+D dataset, models equipped with AMR showed a massive leap in robustness, improving accuracy against PGD20 attacks by 43.77% over previous multimodal defense mechanisms. Quantitative analysis of the attention weights confirmed that the model consistently assigned higher average weights to skeletal features when under attack, validating the intuition that the model learns to "trust" the more robust modality.
Conclusion and Future Work
The Attention-based Modality Reweighter provides a significant step forward in securing multimodal action recognition. By dynamically re-weighting features based on their reliability, AMR effectively mitigates the impact of adversarial perturbations in the RGB stream. While the method introduces a slight robustness-accuracy trade-off on clean data, the gains in adversarial settings are substantial. Future research will focus on extending AMR to Video Transformer backbones and exploring its efficacy against black-box attacks where the model's internal parameters are unknown to the attacker.
Summary updated