Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillation

Yuzheng Wang, Zhaoyu Chen, Dingkang Yang, Pinxue Guo, Kaixun Jiang, Wenqiang Zhang, Lizhe Qi

TL;DR

This paper introduces Data-Free Adversarial Robustness Distillation (DFARD), a novel framework for training small, robust neural networks without access to any original training data. By dynamically adjusting distillation temperatures and adaptively balancing generator losses, the authors achieve state-of-the-art robustness in data-private scenarios while drastically reducing the need for manual hyperparameter tuning.

What Problem Does This Paper Address?

Deep learning models are notoriously vulnerable to adversarial attacks—tiny, intentional perturbations that flip a model's prediction. While Adversarial Training (AT) can defend against these, it is computationally expensive and often requires large model capacities. Adversarial Robustness Distillation (ARD) offers a solution by transferring the "robustness knowledge" from a large, pre-trained teacher model to a smaller, deployable student model.

However, existing ARD methods have a major Achilles' heel: they assume the original training data is available. In high-security or privacy-sensitive fields like medical imaging or financial analysis, this data is often strictly private. The authors propose DFARD to bridge this gap, enabling the creation of robust "tiny" models "out of thin air"—using only a pre-trained teacher and a generator to synthesize training samples.

Figure 1: Diagrams of (a) Knowledge Distillation (KD), (b) Adversarial Robustness Distillation (ARD), and (c) Data-Free Adversarial Robustness Distillation (DFARD). S and T represent the student and the teacher network respectively. Fs and Ft represent the search spaces. xori and xgen is the original and generated data. xadv is the adversarial examples.

Methods

The core challenge of DFARD is that synthetic data carries less information than real data. To maximize the efficiency of this limited information, the authors introduce two key components:

  1. Interactive Temperature Adjustment (ITA): Inspired by human curriculum learning, ITA creates an "easy-to-hard" training process. In distillation, a higher "temperature" ($\tau$) softens the teacher's output, making it easier for a novice student to learn. As the student improves, ITA dynamically lowers the temperature to provide sharper, more challenging objectives.
  2. Adaptive Generator Balance (AGB): Data-free generators typically struggle to balance two goals: making data look like the target domain (similarity) and making it useful for the student (information content). AGB adaptively weights these competing losses based on the teacher's confidence, ensuring the generator doesn't "over-fit" to simple patterns that lack educational value for the student.

These modules are "plug-and-play," meaning they can be integrated into existing data-free pipelines to improve performance without complex manual tuning.

Datasets

The researchers evaluated their method on two standard computer vision benchmarks:

  • CIFAR-10: 10 classes of 32x32 images.
  • CIFAR-100: 100 classes of 32x32 images.

They used various WideResNet (WRN) architectures as teachers and ResNet-18 or MobileNet-V2 as students to test the method's adaptability across different model capacities.

Key Insights and Ideas

  • The Information Bound: The authors theoretically demonstrate that the "information upper bound" for knowledge transfer is lower in data-free tasks than in data-driven ones. This makes efficient transfer strategies like ITA not just helpful, but essential.
  • Easy-to-Hard is Key: Through "toy experiments," the authors proved that a step-decreasing temperature strategy (starting easy, getting harder) consistently outperforms fixed or increasing temperature strategies.
Figure 2: A toy experiment about the effect of different temperatures and simple verification of the easy-to-hard process. (a) and (b) show the impact of using different fixed temperatures for student performance on ARD and DFARD, respectively. (c) shows simple step temperature strategies, which means the trend of varying difficulty changes in the learning objective. (d) shows the performance comparison of students under these strategies.
  • The Confidence Paradox: High teacher confidence in synthetic data actually reduces the information entropy of the soft labels. AGB acts as a regulator, preventing the generator from producing "too-perfect" samples that have low information value.
Figure 3: Diagrams of teacher’s soft labels. As the tempera- ture increases, the distance between the teacher’s and naive student’s prediction distributions decreases, and the difficulty of the learning objectives decreases.

Key Results

  • Superior Robustness: On CIFAR-10, the DFARD method improved average adversarial robustness by up to 1.98% over the best existing data-free baselines (like CMI or DeepInv).
  • Computational Efficiency: Because the parameters are adaptive, the method eliminates the need for exhaustive grid searches for hyperparameters. The authors report that their adaptive calculations add only 0.002% (about 1.4 seconds) to the total training time, whereas manual tuning can take upwards of 20 hours.
  • Teacher Adaptability: The method remains stable even when the capacity gap between the teacher and student is large, avoiding the "robustness saturation" that plagues other distillation techniques.

Limitations and Future Work

While the paper marks a significant step forward, a few areas remain for exploration:

  • Performance Gap: There is still a noticeable performance gap between data-free methods and standard ARD that uses real data.
  • Scaling to High Resolution: The experiments were conducted on 32x32 images. Scaling DFARD to high-resolution datasets (like ImageNet) remains a significant challenge for data-free generation due to the increased complexity of the search space.
  • Generator Diversity: Future work could explore more diverse generator architectures to further increase the information entropy of synthesized samples.

Summary updated

Abstract

Adversarial Robustness Distillation (ARD) is a promising task to solve the issue of limited adversarial robustness of small capacity models while optimizing the expensive computational costs of Adversarial Training (AT). Despite the good robust performance, the existing ARD methods are still impractical to deploy in natural high-security scenes due to these methods rely entirely on original or publicly available data with a similar distribution. In fact, these data are almost always private, specific, and distinctive for scenes that require high robustness. To tackle these issues, we propose a challenging but significant task called Data-Free Adversarial Robustness Distillation (DFARD), which aims to train small, easily deployable, robust models without relying on data. We demonstrate that the challenge lies in the lower upper bound of knowledge transfer information, making it crucial to mining and transferring knowledge more efficiently. Inspired by human education, we design a plug-and-play Interactive Temperature Adjustment (ITA) strategy to improve the efficiency of knowledge transfer and propose an Adaptive Generator Balance (AGB) module to retain more data information. Our method uses adaptive hyperparameters to avoid a large number of parameter tuning, which significantly outperforms the combination of existing techniques. Meanwhile, our method achieves stable and reliable performance on multiple benchmarks.
BibTeX
@inproceedings{wang2024out,
  author = {Yuzheng Wang and Zhaoyu Chen and Dingkang Yang and Pinxue Guo and Kaixun Jiang and Wenqiang Zhang and Lizhe Qi},
  title = {{Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i6.28390},
}
Figure