Adversarial Robustness in RGB-Skeleton Action Recognition: Leveraging Attention Modality Reweighter

Chao Liu, Xin Liu, Zitong Yu, Yonghong Hou, Huanjing Yue, Jing-Yu Yang

TL;DR

This paper tackles the vulnerability of RGB-skeleton action recognition models to adversarial attacks. They introduce the Attention-based Modality Reweighter (AMR), a novel module that dynamically adjusts the influence of RGB and skeleton data streams based on their inherent robustness, significantly improving defense against attacks.

What Problem Does This Paper Address?

Deep neural networks (DNNs) are powerful tools for computer vision, but they are susceptible to adversarial attacks. These attacks involve subtly altering input data (imperceptible to humans) to cause misclassification, posing a significant risk in security-critical applications like autonomous driving and medical diagnostics. While research has explored adversarial robustness in unimodal domains (e.g., image recognition), the robustness of multimodal systems, specifically RGB-skeleton action recognition, remains underexplored. This paper aims to bridge this gap by investigating and enhancing the adversarial robustness of these multimodal models.

Methods

The core of this work is the Attention-based Modality Reweighter (AMR). The authors first empirically observe that skeleton data is inherently more robust to adversarial attacks than RGB data. Motivated by this, AMR uses an attention mechanism to dynamically re-weight the features extracted from the RGB and skeleton modalities. This allows the model to learn more robust features by giving more importance to the more resilient skeleton data and less to the vulnerable RGB data when under attack. AMR is designed to be a plug-and-play module, easily integrated into existing multimodal architectures. They also propose a novel loss function that incorporates auxiliary predictions from AMR at multiple layers, acting as deep supervision to further enhance robustness. The training strategy employed is adversarial training, where both natural and adversarial samples are used.

Datasets

The paper uses two established datasets for evaluation:

  • NTU-RGB+D [33]: A large-scale dataset for 3D human activity analysis, containing RGB videos and corresponding skeleton data.
  • iMiGUE [22]: A dataset focused on micro-gesture understanding and emotion analysis, also providing RGB videos and skeleton data.

Key Insights and Ideas

  • Skeleton Modality is Inherently More Robust: The paper provides strong empirical evidence that skeleton data is significantly more resilient to adversarial perturbations compared to RGB data. This is a crucial observation that underpins their proposed method.
  • Multimodal Fusion Can Degrade Robustness: Counterintuitively, simply fusing RGB and skeleton data does not automatically enhance robustness and can even decrease it when one modality is attacked. This highlights the need for intelligent fusion strategies.
  • Dynamic Reweighting is Key: By using an attention mechanism to re-weight modalities based on their robustness, the model can effectively leverage the strengths of each modality and mitigate the weaknesses of the more vulnerable ones.
  • Deep Supervision for Robustness: Incorporating auxiliary classification predictions from AMR at multiple layers acts as a form of deep supervision, forcing the network to learn robust features throughout its architecture.

Key Results

The proposed AMR method demonstrates significant improvements in adversarial robustness:

  • On the NTU-RGB+D 60 dataset, AMR achieved a 43.77% improvement against PGD20 attacks compared to SOTA methods.
  • The method effectively balances the robustness discrepancies between RGB and skeleton modalities.
  • Experimental results show that AMR consistently outperforms baseline methods and other multimodal defense strategies across various attack types (FGSM, PGD10, CW) and datasets.
  • The paper also shows that the effectiveness of AMR can be influenced by the number of AMR modules inserted and the hyperparameter $\lambda$ controlling the loss function balance.

Limitations and Future Work

  • Dataset Size Impact: The paper notes that for larger datasets like NTU-RGB+D, more AMR modules might be beneficial, whereas for smaller datasets like iMiGUE, a single AMR module might suffice or even be optimal. This suggests a potential need for adaptive strategies for determining the number of AMR modules.
  • Hyperparameter Tuning: The impact of the hyperparameter $\lambda$ on the trade-off between clean and robust accuracy is explored, indicating that careful tuning is necessary for optimal performance on different datasets.
  • Computational Cost: While AMR is presented as plug-and-play, the addition of attention mechanisms and auxiliary loss functions might introduce some computational overhead, which is not extensively detailed.

Future work could explore more sophisticated attention mechanisms, investigate the optimal placement and number of AMR modules for various architectures and datasets, and further analyze the trade-offs between robustness and computational efficiency.

Summary updated

Abstract

Deep neural networks (DNNs) have been applied in many computer vision tasks and achieved state-of-the-art (SOTA) performance. However, misclassification will occur when DNNs predict adversarial examples which are created by adding human-imperceptible adversarial noise to natural examples. This limits the application of DNN in security-critical fields. In order to enhance the robustness of models, previous research has primarily focused on the unimodal domain, such as image recognition and video understanding. Although multi-modal learning has achieved advanced performance in various tasks, such as action recognition, research on the robustness of RGB-skeleton action recognition models is scarce. In this paper, we systematically investigate how to improve the robustness of RGB-skeleton action recognition models. We initially conducted empirical analysis on the robustness of different modalities and observed that the skeleton modality is more robust than the RGB modality. Motivated by this observation, we propose the \formatword{A}ttention-based \formatword{M}odality \formatword{R}eweighter (\formatword{AMR}), which utilizes an attention layer to re-weight the two modalities, enabling the model to learn more robust features. Our AMR is plug-and-play, allowing easy integration with multimodal models. To demonstrate the effectiveness of AMR, we conducted extensive experiments on various datasets. For example, compared to the SOTA methods, AMR exhibits a 43.77\% improvement against PGD20 attacks on the NTU-RGB+D 60 dataset. Furthermore, it effectively balances the differences in robustness between different modalities.
BibTeX
@inproceedings{liu2024adversarial,
  author = {Chao Liu and Xin Liu and Zitong Yu and Yonghong Hou and Huanjing Yue and Jing-Yu Yang},
  title = {{Adversarial Robustness in RGB-Skeleton Action Recognition: Leveraging Attention Modality Reweighter}},
  booktitle = {IJCB},
  year = {2024},
  doi = {10.1109/ijcb62174.2024.10744468},
}
Figure