Abstract
Large-scale deep learning models have achieved great performance based on large-scale datasets. Moreover, the existing Adversarial Training (AT) can further improve the robustness of these large models. However, these large models are difficult to deploy to mobile devices, and the effect of AT on small models is very limited. In addition, the data privacy issue (e.g., face data and diagnosis report) may lead to the original data being unavailable, which relies on data-free knowledge distillation technology for training. To tackle these issues, we propose a challenging novel task called Data-Free Adversarial Robustness Distillation (DFARD), which tries to train small, easily deployable, robust models without relying on the original data. We find the combination of existing techniques resulted in degraded model performance due to fixed training objectives and scarce information content. First, an interactive strategy is designed for more efficient knowledge transfer to find more suitable training objectives at each epoch. Then, we explore an adaptive balance method to suppress information loss and obtain more data information than previous methods. Experiments show that our method improves baseline performance on the novel task.
Introduction
Deep learning models are notoriously vulnerable to adversarial attacks—tiny, invisible perturbations to an image that can trick a model into misclassifying it. While we can train models to be robust, the process is computationally expensive and usually results in massive models that are too slow for mobile devices or drones. Adversarial Robustness Distillation (ARD) was invented to solve this by "distilling" the security knowledge of a large, robust teacher model into a smaller student model. However, existing ARD methods require access to the original training data. In high-security fields like medical imaging or facial recognition, that data is often private and unavailable. This paper tackles the challenge of building robust small models when you have a robust teacher, but zero access to the original data.
The DFARD Framework
The authors propose a "Data-Free" approach where a Generator network creates synthetic images to train the student. This framework, called Data-Free Adversarial Robustness Distillation (DFARD), bridges the gap between model efficiency, robustness, and data privacy.
As illustrated in the figure above, while traditional KD lacks robustness and standard ARD requires sensitive data, DFARD successfully navigates both constraints by synthesizing training data from the teacher model's knowledge.
Methodology
The transition to a data-free setting introduces an "Information Bound" challenge. Because robust models are mathematically "smoother," the amount of information that can be transferred via synthetic data is lower than in standard settings. To overcome this, the researchers introduced two key innovations:
- Interactive Temperature Adjustment (ITA): In distillation, "temperature" controls how "soft" or "hard" the teacher's advice is. ITA dynamically adjusts this temperature based on the student's current performance, creating an "easy-to-hard" curriculum. Starting with high-temperature (easy) examples and moving to low-temperature (hard) examples significantly boosts final robustness.
- Adaptive Generator Balance (AGB): Generating synthetic data involves a balance between making images look like the original domain and making them informative for the student. AGB uses adaptive hyperparameters to balance these goals automatically, removing the need for manual tuning.
Experimental analysis of temperature strategies confirms that a "step decrease" approach (Strategy C in the figure) consistently yields the highest robust accuracy under AutoAttack (AA) for both ARD and DFARD settings.
Key Results
The proposed DFARD method consistently outperformed existing data-free distillation techniques across standard computer vision benchmarks like CIFAR-10 and CIFAR-100:
- Robustness Gains: On CIFAR-10, the method improved average adversarial robustness by approximately 1.98% over the previous best-performing baseline.
- Stability: While other methods often failed on the more complex CIFAR-100 dataset, DFARD remained stable, outperforming the next best method by 1.03%.
- Efficiency: By avoiding manual hyperparameter tuning, the total synthesis time was drastically reduced. For CIFAR-100, DFARD took about 5.85 hours, whereas competing methods required over 100 hours of tuning to reach peak results.
Conclusion
DFARD represents a significant step forward in making robust AI deployable in privacy-sensitive environments. By leveraging interactive learning and adaptive balancing, the framework allows small models to inherit the security of large teachers without ever seeing the original data. Future work will explore incorporating multi-teacher distillation to further close the gap between data-free and data-driven robustness.
Summary updated