TL;DR
This paper tackles the vulnerability of RGB-skeleton action recognition models to adversarial attacks. They introduce the Attention-based Modality Reweighter (AMR), a novel module that dynamically adjusts the influence of RGB and skeleton data streams based on their inherent robustness, significantly improving defense against attacks.
What Problem Does This Paper Address?
Deep neural networks (DNNs) are powerful tools for computer vision, but they are susceptible to adversarial attacks. These attacks involve subtly altering input data (imperceptible to humans) to cause misclassification, posing a significant risk in security-critical applications like autonomous driving and medical diagnostics. While research has explored adversarial robustness in unimodal domains (e.g., image recognition), the robustness of multimodal systems, specifically RGB-skeleton action recognition, remains underexplored. This paper aims to bridge this gap by investigating and enhancing the adversarial robustness of these multimodal models.
Methods
The core of this work is the Attention-based Modality Reweighter (AMR). The authors first empirically observe that skeleton data is inherently more robust to adversarial attacks than RGB data. Motivated by this, AMR uses an attention mechanism to dynamically re-weight the features extracted from the RGB and skeleton modalities. This allows the model to learn more robust features by giving more importance to the more resilient skeleton data and less to the vulnerable RGB data when under attack. AMR is designed to be a plug-and-play module, easily integrated into existing multimodal architectures. They also propose a novel loss function that incorporates auxiliary predictions from AMR at multiple layers, acting as deep supervision to further enhance robustness. The training strategy employed is adversarial training, where both natural and adversarial samples are used.
Datasets
The paper uses two established datasets for evaluation:
- NTU-RGB+D [33]: A large-scale dataset for 3D human activity analysis, containing RGB videos and corresponding skeleton data.
- iMiGUE [22]: A dataset focused on micro-gesture understanding and emotion analysis, also providing RGB videos and skeleton data.
Key Insights and Ideas
- Skeleton Modality is Inherently More Robust: The paper provides strong empirical evidence that skeleton data is significantly more resilient to adversarial perturbations compared to RGB data. This is a crucial observation that underpins their proposed method.
- Multimodal Fusion Can Degrade Robustness: Counterintuitively, simply fusing RGB and skeleton data does not automatically enhance robustness and can even decrease it when one modality is attacked. This highlights the need for intelligent fusion strategies.
- Dynamic Reweighting is Key: By using an attention mechanism to re-weight modalities based on their robustness, the model can effectively leverage the strengths of each modality and mitigate the weaknesses of the more vulnerable ones.
- Deep Supervision for Robustness: Incorporating auxiliary classification predictions from AMR at multiple layers acts as a form of deep supervision, forcing the network to learn robust features throughout its architecture.
Key Results
The proposed AMR method demonstrates significant improvements in adversarial robustness:
- On the NTU-RGB+D 60 dataset, AMR achieved a 43.77% improvement against PGD20 attacks compared to SOTA methods.
- The method effectively balances the robustness discrepancies between RGB and skeleton modalities.
- Experimental results show that AMR consistently outperforms baseline methods and other multimodal defense strategies across various attack types (FGSM, PGD10, CW) and datasets.
- The paper also shows that the effectiveness of AMR can be influenced by the number of AMR modules inserted and the hyperparameter $\lambda$ controlling the loss function balance.
Limitations and Future Work
- Dataset Size Impact: The paper notes that for larger datasets like NTU-RGB+D, more AMR modules might be beneficial, whereas for smaller datasets like iMiGUE, a single AMR module might suffice or even be optimal. This suggests a potential need for adaptive strategies for determining the number of AMR modules.
- Hyperparameter Tuning: The impact of the hyperparameter $\lambda$ on the trade-off between clean and robust accuracy is explored, indicating that careful tuning is necessary for optimal performance on different datasets.
- Computational Cost: While AMR is presented as plug-and-play, the addition of attention mechanisms and auxiliary loss functions might introduce some computational overhead, which is not extensively detailed.
Future work could explore more sophisticated attention mechanisms, investigate the optimal placement and number of AMR modules for various architectures and datasets, and further analyze the trade-offs between robustness and computational efficiency.
Summary updated