Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillation

Yuzheng Wang, Zhaoyu Chen, Dingkang Yang, Pinxue Guo, Kaixun Jiang, Wenqiang Zhang, Lizhe Qi

Abstract

Adversarial Robustness Distillation (ARD) is a promising task to solve the issue of limited adversarial robustness of small capacity models while optimizing the expensive computational costs of Adversarial Training (AT). Despite the good robust performance, the existing ARD methods are still impractical to deploy in natural high-security scenes due to these methods rely entirely on original or publicly available data with a similar distribution. In fact, these data are almost always private, specific, and distinctive for scenes that require high robustness. To tackle these issues, we propose a challenging but significant task called Data-Free Adversarial Robustness Distillation (DFARD), which aims to train small, easily deployable, robust models without relying on data. We demonstrate that the challenge lies in the lower upper bound of knowledge transfer information, making it crucial to mining and transferring knowledge more efficiently. Inspired by human education, we design a plug-and-play Interactive Temperature Adjustment (ITA) strategy to improve the efficiency of knowledge transfer and propose an Adaptive Generator Balance (AGB) module to retain more data information. Our method uses adaptive hyperparameters to avoid a large number of parameter tuning, which significantly outperforms the combination of existing techniques. Meanwhile, our method achieves stable and reliable performance on multiple benchmarks.

Introduction

Deep learning models are notoriously fragile; a tiny, invisible perturbation to an image (an "adversarial attack") can easily fool them. While Adversarial Training (AT) makes models robust, it is computationally expensive and often fails to work well on small, "low-capacity" models like those found on smartphones or IoT devices.

To fix this, researchers use Adversarial Robustness Distillation (ARD), where a large, robust "teacher" model trains a small "student." However, existing ARD methods assume access to the original training data. In real-world scenarios, such data is often private (e.g., medical records or facial data) or simply unavailable. This work introduces Data-Free Adversarial Robustness Distillation (DFARD), a framework for building robust small models when one has the teacher model but zero access to the original data.

Comparison of standard Knowledge Distillation (KD), Adversarial Robustness Distillation (ARD), and the proposed Data-Free Adversarial Robustness Distillation (DFARD). DFARD successfully addresses both data unavailability and adversarial robustness.

Methodology

The challenge in DFARD lies in the lower "information upper bound" when using synthetic data instead of real samples. To overcome this, the authors propose two main components:

Interactive Temperature Adjustment (ITA)

In knowledge distillation, "temperature" ($\tau$) controls the smoothness of the teacher's soft labels. High temperatures make the task easier for the student by revealing more of the teacher's distribution, while low temperatures make it harder. Inspired by human education, ITA dynamically adjusts the temperature based on the student's current performance, creating an "easy-to-hard" curriculum. As shown in experimental results, a "step decrease" strategy (starting with high temperature and moving to low) significantly outperforms fixed-temperature strategies.

Analysis of temperature strategies: (a, b) show robust accuracy vs. temperature T; (c) illustrates the different temperature strategies over training epochs; (d) demonstrates that the 'step decrease' strategy (C) yields the highest robust accuracy.

Adaptive Generator Balance (AGB)

Since real data is absent, a generator network must create synthetic images. Generators often struggle to balance between making images the teacher is confident about and images that contain diverse information. AGB uses an adaptive weight ($\lambda$) to balance these goals automatically, ensuring the synthetic data is both high-quality and informative for the student model.

Key Results

The proposed DFARD method consistently outperformed existing data-free methods across multiple benchmarks (CIFAR-10, CIFAR-100) and attack types (FGSM, PGD, AutoAttack):

  • Robustness Gains: On CIFAR-10 with a ResNet-18 student, the method achieved 32.14% accuracy under AutoAttack (AA), outperforming the previous best data-free method (DeepInv) by over 4%.
  • Stability: Unlike some baselines that performed well on CIFAR-10 but failed on CIFAR-100, DFARD remained stable and effective across both.
  • Efficiency: By utilizing adaptive hyperparameters, DFARD eliminates the need for massive manual grid searches. For CIFAR-100, DFARD required approximately 5.85 hours, whereas competing methods like CMI took nearly 78 hours when accounting for hyperparameter tuning.

Conclusion

DFARD significantly closes the gap between data-free and data-driven adversarial robustness distillation. By employing an "easy-to-hard" learning strategy and adaptive generator balancing, the framework provides a practical path for deploying robust models in high-security, data-private environments. Future work may involve scaling this approach to larger datasets like ImageNet or exploring applications in Natural Language Processing.

Summary updated

Abstract

Adversarial Robustness Distillation (ARD) is a promising task to solve the issue of limited adversarial robustness of small capacity models while optimizing the expensive computational costs of Adversarial Training (AT). Despite the good robust performance, the existing ARD methods are still impractical to deploy in natural high-security scenes due to these methods rely entirely on original or publicly available data with a similar distribution. In fact, these data are almost always private, specific, and distinctive for scenes that require high robustness. To tackle these issues, we propose a challenging but significant task called Data-Free Adversarial Robustness Distillation (DFARD), which aims to train small, easily deployable, robust models without relying on data. We demonstrate that the challenge lies in the lower upper bound of knowledge transfer information, making it crucial to mining and transferring knowledge more efficiently. Inspired by human education, we design a plug-and-play Interactive Temperature Adjustment (ITA) strategy to improve the efficiency of knowledge transfer and propose an Adaptive Generator Balance (AGB) module to retain more data information. Our method uses adaptive hyperparameters to avoid a large number of parameter tuning, which significantly outperforms the combination of existing techniques. Meanwhile, our method achieves stable and reliable performance on multiple benchmarks.
BibTeX
@inproceedings{wang2024out,
  author = {Yuzheng Wang and Zhaoyu Chen and Dingkang Yang and Pinxue Guo and Kaixun Jiang and Wenqiang Zhang and Lizhe Qi},
  title = {{Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i6.28390},
}
Figure