Global-Local Characteristic Excited Cross-Modal Attacks from Images to Videos

Ruikui Wang, Yuanfang Guo, Yunhong Wang

TL;DR

This paper introduces a novel cross-modal attack method that generates adversarial examples for video recognition models using only image-based substitute models. By considering both global inter-frame interactions and local temporal correlations within videos, their approach significantly boosts attack transferability, outperforming existing methods and even some that use video models as substitutes.

What Problem Does This Paper Address?

The security of deep neural networks (DNNs) is a growing concern, especially in safety-critical applications like autonomous driving and facial recognition. Adversarial examples, subtle perturbations to input data, can easily fool these models. While adversarial attacks have been extensively studied for images, their application to videos is more challenging due to the temporal dimension. A key challenge is cross-modal transferability: generating adversarial examples using a simpler, more accessible model (like an image model) to attack a more complex target model (like a video model). Existing methods that attempt this often treat videos as mere collections of independent frames, ignoring crucial temporal information, which limits their effectiveness. This paper aims to bridge this gap by developing a more sophisticated cross-modal attack that leverages the unique characteristics of video data.

Methods

The authors propose a two-pronged approach to enhance cross-modal adversarial attacks on videos:

  • Global Interaction Enhancement (GIE): Instead of treating each video frame independently, GIE models the entire video clip as a convex hull spanned by its frames. This allows for "inter-frame interactions," where gradients from attacking one frame are influenced by other frames. This is achieved by minimizing the cosine similarity between intermediate features of "benign" points on the convex hull and their corresponding "adversarial" points. This global perspective aims to generate more diverse and stronger attack gradients.
  • Local Correlation Disturbance (LCD): Videos inherently possess strong temporal correlations between adjacent frames, which are vital for action recognition. LCD aims to disrupt these correlations. The authors demonstrate that image models and video models capture similar temporal local correlations. By optimizing adversarial perturbations to decrease the similarity between intermediate features of adjacent adversarial frames, LCD makes it harder for target video models to leverage temporal cues, thus increasing attack success.

The final adversarial perturbation is optimized by combining the losses from GIE and LCD.

Datasets

  • Training/Substitute Models: Image models pre-trained on ImageNet were used as substitute models. Specifically, AlexNet, ResNet-101, SqueezeNet, and VGG-16 were employed.
  • Evaluation: Two video recognition datasets were used for evaluation:
  • UCF-101: Contains 13,320 videos across 101 action categories.
  • Kinetics-400: A larger dataset with approximately 240,000 videos across 400 action categories.

The target black-box video models used for evaluation included Non-local (NL), SlowFast, and TPN architectures.

Key Insights and Ideas

  • Videos are more than just image sequences: The paper strongly emphasizes that ignoring temporal dynamics in videos leads to suboptimal adversarial attacks.
  • Global and Local perspectives are complementary: The GIE (global) and LCD (local) components work together effectively, with GIE boosting transferability and LCD exploiting model vulnerabilities.
  • Image models can be surprisingly effective attackers: The success of their cross-modal attack, even surpassing some video-model-based attacks, highlights the potential for simpler models to generate potent adversarial examples for complex video models.
  • Temporal correlations are a key vulnerability: Disrupting these correlations, which are learned by both image and video models, proves to be a powerful attack strategy.
  • Convex hulls for inter-frame interaction: Using convex hulls to represent video clips is an elegant way to introduce global interactions and generate richer gradients.

Key Results

The proposed method, GIE+LCD, consistently outperformed baseline methods and the existing cross-modal attack method I2V across various substitute image models and target video models.

  • Significant ASR improvements: On UCF-101, the method achieved average attack success rates (ASR) improved by up to 10.1% compared to baselines. On Kinetics-400, improvements reached up to 13.6%.
  • Outperforming I2V: For example, using VGG-16 as a substitute, their method surpassed I2V by 10.07% on UCF-101 and 13.08% on Kinetics-400.
  • Competitive with video-based attacks: In some scenarios, their cross-modal attack even achieved comparable or better results than methods using video models as substitutes.
  • Model ensemble benefits: Using an ensemble of substitute models (DenseNet-121 and ResNet-101) further boosted performance, achieving an ASR of 85.50% on Kinetics-400 against stronger baselines.

Limitations and Future Work

  • Computational Cost: While the method is more economical than using video models as substitutes, generating adversarial examples still involves significant computation, especially with larger video clips or more complex interactions. The paper mentions sampling points from the convex hull and using lookup tables to mitigate this.
  • Parameter Sensitivity: The effectiveness of LCD is sensitive to the hyperparameter $\lambda$, which balances the GIE and LCD losses. The paper suggests that smaller $\lambda$ values generally yield better results.
  • Model-Specific Vulnerabilities: The paper notes that the effectiveness of LCD can vary depending on the linearity of correlations captured by different image and video model pairs. For instance, it was less effective when using SqueezeNet on Kinetics-400, possibly due to more linear correlations between SqueezeNet and the target video model.
  • Future Work: The authors suggest exploring the robustness of video recognition models and developing defense algorithms against common attacks. They also hint at further enhancements through model ensembling.

Summary updated

Abstract

The transferability of adversarial examples is the key property in practical black-box scenarios. Currently, numerous methods improve the transferability across different models trained on the same modality of data. The investigation of generating video adversarial examples with imagebased substitute models to attack the target video models, i.e., cross-modal transferability of adversarial examples, is rarely explored. A few works on cross-modal transferability directly apply image attack methods for each frame and no factors especial for video data are considered, which limits the cross-modal transferability of adversarial examples. In this paper, we propose an effective cross-modal attack method which considers both the global and local characteristics of video data. Firstly, from the global perspective, we introduce inter-frame interaction into attack process to induce more diverse and stronger gradients rather than perturb each frame separately. Secondly, from the local perspective, we disrupt the inherently local correlation of frames within a video, which prevents black-box video model from capturing valuable temporal clues. Extensive experiments on the UCF-101 and Kinetics-400 validate the proposed method significantly improves cross-modal transferability and even surpasses strong baseline using video models as substitute model. Our source codes are available at https://github.com/lwmming/Cross-Modal-Attack.
BibTeX
@inproceedings{wang2023global,
  author = {Ruikui Wang and Yuanfang Guo and Yunhong Wang},
  title = {{Global-Local Characteristic Excited Cross-Modal Attacks from Images to Videos}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i2.25362},
}
Figure