Safe Distillation Box

Jingwen Ye, Yining Mao, Jie Song, Xinchao Wang, Cheng Jin, Mingli Song

Abstract

Knowledge distillation (KD) has recently emerged as a powerful strategy to transfer knowledge from a pre-trained teacher model to a lightweight student, and has demonstrated its unprecedented success over a wide spectrum of applications. In spite of the encouraging results, the KD process per se poses a potential threat to network ownership protection, since the knowledge contained in network can be effortlessly distilled and hence exposed to a malicious user. In this paper, we propose a novel framework, termed as Safe Distillation Box (SDB), that allows us to wrap a pre-trained model in a virtual box for intellectual property protection. Specifically, SDB preserves the inference capability of the wrapped model to all users, but precludes KD from unauthorized users. For authorized users, on the other hand, SDB carries out a knowledge augmentation scheme to strengthen the KD performances and the results of the student model. In other words, all users may employ a model in SDB for inference, but only authorized users get access to KD from the model. The proposed SDB imposes no constraints over the model architecture, and may readily serve as a plug-and-play solution to protect the ownership of a pre-trained network. Experiments across various datasets and architectures demonstrate that, with SDB, the performance of an unauthorized KD drops significantly while that of an authorized gets enhanced, demonstrating the effectiveness of SDB.

Introduction

The rapid advancement of deep learning has led to the development of massive, high-performance models. To deploy these models on resource-constrained devices like mobile phones, Knowledge Distillation (KD) is frequently employed to transfer expertise from a large "teacher" model to a smaller "student" model. However, this utility comes with a significant security risk: intellectual property (IP) theft. Malicious actors can use a model's public API to distill its knowledge into their own proprietary models without ever accessing the original source code or weights.

Conceptual overview of the Safe Distillation Box (SDB) framework, illustrating the differential treatment of authorized and unauthorized users.

To combat this, researchers have introduced the Safe Distillation Box (SDB). SDB acts as a virtual protective layer that allows a model to function normally for inference while selectively blocking unauthorized distillation attempts.

Methodology

The SDB framework employs a three-pronged strategy to secure the teacher model:

  1. Key Embedding: The model is trained to recognize a secret "key"—a specific pattern of random noise. When this key is combined with input data (creating "proxy data"), it serves as a cryptographic handshake to unlock the model's full teaching potential.
  2. Knowledge Disturbance (The Lock): For standard inputs lacking the key, the model is optimized to produce "disturbed" logits. While the final classification remains accurate (maintaining utility), the underlying probability distribution is scrambled. This makes the output a poor training signal for unauthorized student models.
  3. Knowledge Preservation & Augmentation (The Key): When the authorized key is present, the model enters "Knowledge Preservation" mode, providing high-fidelity signals for KD. Furthermore, SDB implements a "Knowledge Augmentation" scheme that searches for additional features, often resulting in a student model that outperforms those trained via standard KD.
The SDB technical workflow, highlighting the Key Embedding process and the dual modes of Knowledge Disturbance and Preservation.

Experimental Results

The SDB framework was evaluated using standard benchmarks, including CIFAR-10 and CIFAR-100, across various architectures such as ResNet and MobileNet.

  • Security Effectiveness: On the CIFAR-10 dataset, an unauthorized student model attempting to distill from an SDB-protected ResNet-18 saw its accuracy plummet from 88.06% (standard KD) to just 54.59%.
  • Performance Enhancement: Conversely, authorized users benefited from the augmentation scheme. A CNN student trained from an SDB teacher achieved 89.22% accuracy, surpassing the 88.06% achieved with standard KD.
  • Inference Fidelity: The protection mechanism is highly efficient, with the teacher model losing less than 1% of its original inference accuracy. This ensures the model remains fully functional for its primary production tasks.

Conclusion

The Safe Distillation Box provides a robust, architecture-agnostic solution for protecting model ownership. By decoupling inference utility from distillation capability, SDB ensures that intellectual property remains secure without sacrificing performance for legitimate users. Future work will focus on optimizing SDB for edge deployment and testing its resilience against increasingly sophisticated adaptive attacks.

Summary updated

Abstract

Knowledge distillation (KD) has recently emerged as a powerful strategy to transfer knowledge from a pre-trained teacher model to a lightweight student, and has demonstrated its unprecedented success over a wide spectrum of applications. In spite of the encouraging results, the KD process per se poses a potential threat to network ownership protection, since the knowledge contained in network can be effortlessly distilled and hence exposed to a malicious user. In this paper, we propose a novel framework, termed as Safe Distillation Box (SDB), that allows us to wrap a pre-trained model in a virtual box for intellectual property protection. Specifically, SDB preserves the inference capability of the wrapped model to all users, but precludes KD from unauthorized users. For authorized users, on the other hand, SDB carries out a knowledge augmentation scheme to strengthen the KD performances and the results of the student model. In other words, all users may employ a model in SDB for inference, but only authorized users get access to KD from the model. The proposed SDB imposes no constraints over the model architecture, and may readily serve as a plug-and-play solution to protect the ownership of a pre-trained network. Experiments across various datasets and architectures demonstrate that, with SDB, the performance of an unauthorized KD drops significantly while that of an authorized gets enhanced, demonstrating the effectiveness of SDB.
BibTeX
@inproceedings{ye2022safe,
  author = {Jingwen Ye and Yining Mao and Jie Song and Xinchao Wang and Cheng Jin and Mingli Song},
  title = {{Safe Distillation Box}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i3.20219},
}
Figure