Model Stealing Defense against Exploiting Information Leak through the Interpretation of Deep Neural Nets

Jeonghyun Lee, Sungmin Han, Sangkyun LeeIJCAI 2022 · DOI

TL;DR

This paper introduces DeepDefense, a novel method to protect AI models from "model stealing" attacks that exploit not only prediction outputs but also interpretability information (like attribution maps). DeepDefense uses a misdirection model to subtly alter these outputs, making it harder for attackers to create accurate clones without significantly degrading the original model's functionality or interpretability for legitimate users.

What Problem Does This Paper Address?

Machine learning models, especially those offered as a service, are vulnerable to "model stealing" attacks. Adversaries aim to create a functional clone of a victim model by querying it and using the responses to train their own model. Traditionally, these attacks focused on using the model's class probability outputs. However, recent research has shown that interpretability tools, which provide insights into why a model makes a certain prediction (e.g., attribution maps highlighting important input features), can also be exploited to improve the accuracy of these stolen models. This creates a new attack vector, as interpretability information can be as valuable as gradients for attackers. The challenge is to defend against these more sophisticated attacks that leverage both prediction outputs and interpretability, without compromising the original model's performance or its usefulness for benign users.

Methods

The core of DeepDefense is a "misdirection model." This model acts as an intermediary, receiving user queries and returning outputs that are subtly altered to mislead potential attackers. The key idea is to "misdirect" the critical information attackers use, specifically the gradients of the model's top and bottom parts (which are used by many interpretability methods).

DeepDefense achieves this by:

  • Gradient Misdirection: It orthogonalizes the gradients of the misdirection model's parameters with respect to those of the original victim model. This is done separately for the top and bottom parts of the network, allowing for controlled misdirection.
  • Functionality Preservation: The misdirection model's output must still preserve the order of the top-k softmax probabilities of the original model. This ensures that for legitimate users, the model's predictions remain functionally the same.
  • Interpretability Preservation: The order of values in the attribution map generated by the misdirection model should also match that of the original model. This ensures that the relative importance of input features is maintained, preserving interpretability.

To implement this, the authors formulate an optimization problem. They define an "orthogonalization loss" to achieve gradient misdirection and use penalty functions for functionality and interpretability preservation. They also introduce a strategy to select only the most "sensitive" layers for optimization, reducing computational cost.

Datasets

The paper uses the following datasets for training and evaluating victim models:

  • MNIST
  • KMNIST
  • CIFAR-10
  • Flowers-17
  • CUBS-200

For the attack strategy, they use different transfer-sets for the attacker's data distribution, which is assumed to be different from the victim's:

  • Fashion-MNIST (for MNIST)
  • MNIST (for KMNIST)
  • CIFAR100 (for CIFAR10)
  • ImageNet (for Flowers-17 and CUBS-200)

Key Insights and Ideas

  • Interpretability as a New Attack Surface: The paper highlights that interpretability tools, intended to make AI more transparent, can inadvertently provide attackers with valuable information to steal models more effectively.
  • Misdirection, Not Obfuscation: DeepDefense doesn't just randomly perturb outputs; it strategically misdirects critical gradient information. This is a more nuanced approach than simply making outputs noisy.
  • Balancing Functionality and Security: A key challenge is to protect against attackers without breaking the model for legitimate users. DeepDefense achieves this by preserving the order of probabilities and attribution values, rather than exact values.
  • Layer Sensitivity Matters: Not all layers contribute equally to interpretability. Identifying and focusing on sensitive layers can significantly reduce the computational cost of defense without sacrificing effectiveness.

Key Results

  • Superior Defense Performance: DeepDefense consistently outperformed existing state-of-the-art defenses (Prediction Poisoning, Adaptive Misinformation, Ensemble of Diverse Models) across various datasets and interpreters. The attack models trained against DeepDefense had significantly lower test accuracy, indicating a more successful defense.
  • Consistent Performance: DeepDefense showed consistent defense performance across different query budgets, unlike some other methods that degraded significantly with more queries.
  • Minimal Interpretability Degradation: The paper demonstrates that DeepDefense causes only a small degradation in attribution quality, with no statistically significant difference in interpretability measures compared to the original model. Visual comparisons of attribution maps also showed high similarity.
  • Computational Efficiency: By selecting sensitive layers, DeepDefense achieved a reasonable trade-off between defense performance and computational cost, with runtimes comparable to or better than existing methods.
  • Ablation Study: The study showed that perturbing only softmax outputs (DD-S) was competitive with existing methods, but perturbing only attribution maps (DD-I) was not sufficient on its own, underscoring the importance of addressing both information sources.

Limitations and Future Work

  • Hyperparameter Tuning: The effectiveness of DeepDefense relies on hyperparameters (e.g., a, λ1, λ2, k). While the paper provides experimental settings, optimal tuning might be dataset- and model-dependent.
  • Computational Cost: While optimized by layer selection, the defense mechanism still adds computational overhead compared to an undefended model.
  • Future Work: The authors suggest further research into optimizing the functionality preservation constraint (specifically the value of k) to guarantee high defense performance. They also imply that exploring more sophisticated misdirection strategies could be beneficial.

Summary updated

BibTeX
@inproceedings{lee2022model,
  author = {Jeonghyun Lee and Sungmin Han and Sangkyun Lee},
  title = {{Model Stealing Defense against Exploiting Information Leak through the Interpretation of Deep Neural Nets}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/100},
}
Figure