SwiftThief: Enhancing Query Efficiency of Model Stealing by Contrastive Learning

Jeonghyun Lee, Sungmin Han, Sangkyun LeeIJCAI 2024

Abstract

Model stealing poses a significant threat to the security and intellectual property of proprietary AI models. Current attack methods often suffer from high query costs and performance degradation due to class imbalance in the queried data. We present SwiftThief, a high-efficiency model-stealing framework that leverages contrastive learning on both queried and unqueried data. By employing a novel triple-loss function and a priority sampling strategy, SwiftThief achieves state-of-the-art cloning accuracy while reducing the required query budget by more than 50%.

Introduction

Model stealing involves an adversary creating a "clone" of a black-box victim model by querying it and observing its outputs. This allows attackers to bypass usage fees, steal proprietary logic, or craft adversarial examples. However, high query counts make such attacks expensive and easily detectable. Existing methods typically ignore the potential of unqueried data and struggle with class imbalance, where active learning strategies over-sample frequent classes at the expense of rare ones. SwiftThief addresses these challenges by treating model stealing as a representation learning problem, utilizing unqueried data to build robust features and a smart sampling agent to maintain class balance.

Methodology

SwiftThief's architecture is built upon the SimSiam contrastive learning framework, which learns meaningful representations without requiring negative pairs.

The Triple-Loss Function

SwiftThief optimizes three distinct loss components:

  1. Self-Supervised Contrastive Loss ($L_{self}$): Extracts general features from the pool of unqueried data, allowing the model to learn data representations before labels are obtained.
  2. Soft-Supervised Contrastive Loss ($L_{soft}$): Utilizes the victim model's softmax probabilities as "soft labels." It aligns representations based on the similarity of probability distributions, capturing the "dark knowledge" of inter-class relationships.
  3. Weighted Regularizer ($L_{reg}$): Prevents overfitting on frequent classes by penalizing the sharpness of the representation function around rare class samples.

Priority Sampling for Class Balance

To mitigate class imbalance, SwiftThief employs a dual-sampling strategy. It initially uses entropy-based sampling to identify uncertain instances. When class under-representation is detected, the system switches to Priority Sampling. This method uses Kernel Density Estimation (KDE) in the feature space to identify unqueried samples likely belonging to rare classes, ensuring a balanced training set for the clone model.

SwiftThief System Architecture

Experimental Results

The framework was evaluated against five victim datasets (MNIST, SVHN, GTSRB, CIFAR-10, and EuroSAT) using ImageNet as a surrogate dataset. This setup reflects a realistic scenario where the attacker lacks access to the victim's original training distribution.

Query Efficiency and Accuracy

SwiftThief demonstrated a 2.27x improvement in query efficiency, reaching target accuracy levels with less than half the queries required by baseline methods. At low query budgets (e.g., 2K to 8K queries for MNIST/SVHN), SwiftThief outperformed competitors by factors of 1.2x to 2.6x.

Test Accuracy vs. Query Budget across various datasets

Robustness to Defenses

The model also showed superior resilience against prediction poisoning defenses. While noise-based defenses significantly degraded the performance of other attacks, SwiftThief's contrastive approach maintained high accuracy. Furthermore, the framework remained effective even in hard-label settings where only the top-1 predicted class was available.

Conclusion

SwiftThief significantly advances the state-of-the-art in model stealing by integrating self-supervised learning with intelligent sampling. By effectively utilizing unqueried data and addressing class imbalance, it provides a more potent and efficient threat to proprietary AI models. Future research may explore the integration of explainable AI outputs, such as attribution maps, to further enhance the efficiency of the stealing process.

Summary updated

BibTeX
@inproceedings{lee2024swiftthief,
  author = {Jeonghyun Lee and Sungmin Han and Sangkyun Lee},
  title = {{SwiftThief: Enhancing Query Efficiency of Model Stealing by Contrastive Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure