A Tale of Two Models: Constructing Evasive Attacks on Edge Models

Wei Hao, Aahil Awatramani, Jiayang Hu, Chengzhi Mao, Pin-Chun Chen, Eyal Cidon, Asaf Cidon, Junfeng YangMLSys 2022 · OpenReview

TL;DR

This paper introduces DIVA, a novel adversarial attack that exploits subtle differences between edge-adapted deep learning models and their original, full-precision counterparts. DIVA crafts adversarial examples that fool edge devices while remaining virtually undetectable by the original model, posing a significant threat to deployed AI systems.

What Problem Does This Paper Address?

Deep learning models are increasingly deployed on resource-constrained edge devices (like smartphones and cameras). To make these models fit, techniques like quantization and pruning are used, which can slightly alter their behavior compared to the original, high-precision models. This paper addresses the security vulnerability created by these adaptations: an attacker can craft an input that causes the edge model to misbehave, but this same input would be correctly classified by the original model. This makes detection and debugging of such attacks extremely difficult for system operators, who often rely on the original model for validation.

Methods

The core of DIVA is its exploitation of the "instability" or divergence between the original and adapted models. It uses a differential testing approach, aiming to maximize the prediction loss on the adapted model while minimizing it on the original model.

  • Whitebox DIVA: The attacker has full access to both models. DIVA formulates a joint loss function that balances the prediction errors of both models, then uses an optimization method like Projected Gradient Descent (PGD) to generate adversarial perturbations.
  • Semi-blackbox DIVA: The attacker only has access to the adapted model. They reconstruct a surrogate full-precision model and then apply a modified DIVA attack.
  • Blackbox DIVA: The attacker has no access to model parameters. They reconstruct both a surrogate full-precision model and a surrogate adapted model, then attack using these surrogates.

Datasets

  • ImageNet: Used for training and evaluating models on object recognition tasks.
  • MNIST: Used for visualization of learned representations.
  • PubFig: Used for the face recognition case study.

Key Insights and Ideas

  • Model Adaptation Creates Vulnerabilities: The very process of making models suitable for edge devices introduces subtle differences that can be exploited.
  • Differential Testing for Attacks: Instead of just attacking one model, DIVA leverages the comparison between two similar models to create stealthy attacks.
  • Stealth is Key: The primary goal is to fool the edge device without raising suspicion from the authoritative original model, making detection and debugging a nightmare for operators.
  • Low Cost for High Evasion: DIVA achieves high evasion rates with only a minor drop in attack success compared to standard attacks like PGD.

Key Results

  • High Success Rates: Whitebox DIVA achieved 92.3–97% success in causing adapted models to mispredict while keeping the original model correct. Semi-blackbox DIVA achieved 71.1-96.2% success.
  • Stealthy Attacks: DIVA was significantly better at not affecting the original model compared to PGD. For example, on quantized models, DIVA caused mispredictions on the adapted model while the original model remained correct in 92.3-97% of cases, whereas PGD's success rate in this regard was much lower.
  • Minimal Accuracy Drop: DIVA was only 1.7–3.6% worse than PGD in attacking the adapted model, demonstrating a good trade-off between attack effectiveness and evasion.
  • Face Recognition Case Study: DIVA successfully fooled a quantized VGGFace model, misidentifying Nicolas Cage as Jerry Seinfeld, while the original model correctly identified him.

Limitations and Future Work

  • Blackbox Performance: The blackbox DIVA attack had a lower success rate (30.3–77.2%) compared to whitebox and semi-blackbox versions, likely due to the added complexity of reconstructing surrogate models.
  • Model Architecture Sensitivity: The performance of blackbox DIVA was noted to be worse on smaller models like MobileNet, suggesting potential issues with transferability in less complex architectures.
  • Future Work: The authors suggest that this work opens the door to a new line of research on attacks and defenses that specifically target variations in deployed models. Further exploration of defenses against such differential attacks would be valuable.

Summary updated

Abstract

Full-precision deep learning models are typically too large or costly to deploy on edge devices. To accommodate to the limited hardware resources, models are adapted to the edge using various edge-adaptation techniques, such as quantization and pruning. While such techniques may have a negligible impact on top-line accuracy, the adapted models exhibit subtle differences in output compared to the original model from which they are derived. In this paper, we introduce a new evasive attack, DIVA, that exploits these differences in edge adaptation, by adding adversarial noise to input data that maximizes the output difference between the original and adapted model. Such an attack is particularly dangerous, because the malicious input will trick the adapted model running on the edge, but will be virtually undetectable by the original model, which typically serves as the authoritative model version, used for validation, debugging and retraining. We compare DIVA to a state-of-the-art attack, PGD, and show that DIVA is only 1.7-3.6% worse on attacking the adapted model but 1.9-4.2 times more likely not to be detected by the the original model under a whitebox and semi-blackbox setting, compared to PGD.
BibTeX
@inproceedings{hao2022tale,
  author = {Wei Hao and Aahil Awatramani and Jiayang Hu and Chengzhi Mao and Pin-Chun Chen and Eyal Cidon and Asaf Cidon and Junfeng Yang},
  title = {{A Tale of Two Models: Constructing Evasive Attacks on Edge Models}},
  booktitle = {MLSys},
  year = {2022},
  url = {https://openreview.net/forum?id=-7Zw1njjL-},
}
Figure