Safe Distillation Box

Jingwen Ye, Yining Mao, Jie Song, Xinchao Wang, Cheng Jin, Mingli Song

Introduction

Knowledge Distillation (KD) has emerged as a vital technique in deep learning, enabling the compression of large, high-performance "teacher" models into compact "student" models suitable for resource-constrained environments like mobile devices. However, this accessibility introduces a significant security risk: model theft. An adversary can treat a deployed model as a teacher and distill its knowledge into a clone, effectively stealing the original developer's intellectual property (IP) and research investment.

To address this, researchers from Zhejiang University, NUS, and Fudan University proposed the Safe Distillation Box (SDB). SDB is a security framework designed to "wrap" pre-trained models, allowing standard inference while selectively restricting the ability to perform distillation to authorized users only.

Conceptual overview of the Safe Distillation Box framework, showing how it differentiates between standard prediction requests and distillation attempts by authorized vs. unauthorized users.

The Challenge of Knowledge Theft

The core "knowledge" of a neural network resides not just in its final hard-label predictions, but in its "soft labels"—the probability distributions across all possible classes. These distributions reveal the model's internal understanding of inter-class relationships. By mimicking these soft labels, a student model can achieve performance close to the teacher.

Existing defenses, such as "Nasty Teacher" methods, prevent distillation by providing misleading soft labels to everyone. SDB improves upon this by offering selective access, ensuring that legitimate users can still benefit from distillation while adversaries are thwarted.

Methodology: The Virtual Box

The SDB framework implements a "virtual box" around the teacher model through three primary mechanisms:

  1. Key Embedding: The model owner defines a secret digital "key" (a specific noise pattern). Authorized users must incorporate this key into their training data to create "proxy data." This key serves as a trigger to unlock the model's true knowledge.
  2. Knowledge Disturbance: For standard inputs without the key, the model is trained to behave as a "Nasty Teacher." It outputs intentionally confusing soft labels during distillation requests, which prevents unauthorized student models from learning effectively.
  3. Knowledge Preservation & Augmentation: When the model detects the secret key, it switches to an "open" state. It provides high-fidelity soft labels and employs an augmentation scheme that highlights critical feature relationships. This can actually lead to authorized student models performing better than those distilled from an unprotected teacher.
The technical workflow of SDB, illustrating the key embedding process, the dual-path knowledge space (Disturbance vs. Preservation), and the resulting predictions.

Experimental Evaluation

The authors validated SDB using standard benchmarks including CIFAR-10, CIFAR-100, and Tiny-ImageNet. They tested various architectures, demonstrating that SDB is architecture-agnostic and compatible with models like ResNet, MobileNet, and ShuffleNet.

Key Results

  • Selective Distillation: On CIFAR-10, an unauthorized student model's accuracy dropped from a baseline of 88.06% to just 46.91%. Conversely, authorized users applying the key achieved 88.45% accuracy, successfully recovering the model's full potential.
  • Inference Fidelity: The SDB wrapper does not compromise the teacher's primary function. The protected teacher maintains its original classification accuracy within a ~1% margin for standard inference tasks.
  • Robustness: The framework proved resilient against various adversarial attempts, such as "Random Temperature Attacks" and "Random Key Attacks," maintaining a significant performance gap between authorized and unauthorized distillation.

Conclusion and Future Directions

The Safe Distillation Box provides a robust solution for protecting AI intellectual property without sacrificing utility. By leveraging a trigger-based mechanism inspired by backdoor attacks, it ensures that high-quality distillation remains a privilege for authorized users. Future research will focus on extending SDB to generative models and natural language processing, as well as optimizing it for direct deployment on edge devices.

Summary updated

Abstract

Knowledge distillation (KD) has recently emerged as a powerful strategy to transfer knowledge from a pre-trained teacher model to a lightweight student, and has demonstrated its unprecedented success over a wide spectrum of applications. In spite of the encouraging results, the KD process \emph{per se} poses a potential threat to network ownership protection, since the knowledge contained in network can be effortlessly distilled and hence exposed to a malicious user. In this paper, we propose a novel framework, termed as Safe Distillation Box~(SDB), that allows us to wrap a pre-trained model in a virtual box for intellectual property protection. Specifically, SDB preserves the inference capability of the wrapped model to all users, but precludes KD from unauthorized users. For authorized users, on the other hand, SDB carries out a knowledge augmentation scheme to strengthen the KD performances and the results of the student model. In other words, all users may employ a model in SDB for inference, but only authorized users get access to KD from the model. The proposed SDB imposes no constraints over the model architecture, and may readily serve as a plug-and-play solution to protect the ownership of a pre-trained network. Experiments across various datasets and architectures demonstrate that, with SDB, the performance of an unauthorized KD drops significantly while that of an authorized gets enhanced, demonstrating the effectiveness of SDB.
BibTeX
@inproceedings{ye2022safe,
  author = {Jingwen Ye and Yining Mao and Jie Song and Xinchao Wang and Cheng Jin and Mingli Song},
  title = {{Safe Distillation Box}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i3.20219},
}
Figure