Abstract
Adversarial training is a practical approach for improving the robustness of deep neural networks against adversarial attacks. Although bringing reliable robustness, the performance toward clean examples is negatively affected after adversarial training, which means a trade-off exists between accuracy and robustness. Recently, some studies have tried to use knowledge distillation methods in adversarial training, achieving competitive performance in improving the robustness but the accuracy for clean samples is still limited. In this paper, to mitigate the accuracy-robustness trade-off, we introduce the Multi-Teacher Adversarial Robustness Distillation (MTARD) to guide the model's adversarial training process by applying a strong clean teacher and a strong robust teacher to handle the clean examples and adversarial examples, respectively. During the optimization process, to ensure that different teachers show similar knowledge scales, we design the Entropy-Based Balance algorithm to adjust the teacher's temperature and keep the teachers' information entropy consistent. Besides, to ensure that the student has a relatively consistent learning speed from multiple teachers, we propose the Normalization Loss Balance algorithm to adjust the learning weights of different types of knowledge. A series of experiments conducted on public datasets demonstrate that MTARD outperforms the state-of-the-art adversarial training and distillation methods against various adversarial attacks.
Introduction
Deep Neural Networks (DNNs) are notoriously vulnerable to adversarial attacks—tiny, human-imperceptible perturbations to input data that cause models to make confident, incorrect predictions. While Adversarial Training (AT) is the gold standard for defense, it comes with a "tax": as a model becomes more robust to attacks, its accuracy on normal, "clean" images typically drops. This fundamental trade-off has limited the practical deployment of robust models.
Existing attempts to solve this using Knowledge Distillation (KD)—where a "student" model learns from a "teacher"—often rely on a single robust teacher. However, because of the intrinsic trade-off, even a robust teacher struggles to be perfect at both accuracy and robustness. This paper introduces B-MTARD (Balanced Multi-Teacher Adversarial Robustness Distillation), a framework that uses two specialized teacher models—one optimized for accuracy and one for robustness—to train a single student model.
Methodology: The B-MTARD Framework
The authors propose a "divide-and-conquer" strategy called B-MTARD, featuring two primary innovations to manage the multi-teacher setup. Instead of searching for one "perfect" teacher, it is more effective to use two "specialist" teachers and manage the interference between them.
1. Dual-Teacher Framework
The student learns from a Clean Teacher (high accuracy on normal data) and a Robust Teacher (high accuracy on adversarial data) simultaneously. The clean teacher provides guidance for clean examples, while the robust teacher handles adversarial examples, allowing the student to benefit from specialized expertise in both domains.
2. Entropy-Based Balance (EBB)
Different teacher architectures have different "knowledge scales." The authors mathematically prove that a teacher's knowledge scale is negatively related to its information entropy. EBB automatically adjusts the softmax temperature ($\tau$) of each teacher during training to ensure their information entropy—and thus their influence on the student—remains consistent. Temperature acts as a dynamic tool to equalize the "volume" of knowledge coming from different sources.
3. Normalization Loss Balance (NLB)
Students often find it easier to learn one task (e.g., clean accuracy) over another (e.g., robustness). NLB dynamically adjusts the weights ($w_{nat}, w_{adv}$) of the different loss functions based on the student's relative learning progress, ensuring the student doesn't "overfit" to one teacher while ignoring the other.
Experimental Evaluation
The researchers evaluated their method on three standard computer vision benchmarks: CIFAR-10, CIFAR-100, and Tiny-ImageNet. They used a variety of student architectures, including ResNet-18 and MobileNet-v2, to prove the method's versatility.
Key Results
B-MTARD consistently outperformed state-of-the-art (SOTA) methods like TRADES, ARD, and RSLAD across all datasets:
- Better Balance: On CIFAR-10 (ResNet-18), B-MTARD achieved a clean accuracy of 88.20%, significantly higher than standard Adversarial Training (~84%), while maintaining a robust accuracy of 51.68% against PGD attacks.
- Weighted Robust Accuracy (W-Robust Acc): This metric, which averages clean and robust performance, showed that B-MTARD provides a superior overall "sweet spot" compared to previous distillation methods.
- Resilience: The models remained robust against a wide battery of attacks, including white-box (PGD, CW), black-box (Square Attack), and advanced ensemble attacks like AutoAttack, confirming that the defense does not rely on "obfuscated gradients."
Conclusion
B-MTARD successfully mitigates the accuracy-robustness trade-off by leveraging specialized teachers and dynamic balancing mechanisms. While training with two teachers increases computational overhead, the gains in performance balance are significant. Future work may explore applying this balanced multi-teacher approach to other complex tasks like object detection or natural language processing.
Summary updated