Introduction
Deep Neural Networks (DNNs) are notoriously fragile; tiny, human-imperceptible "adversarial perturbations" can easily trick them into making wrong predictions. To defend against this, researchers typically use Adversarial Training (AT)—training on these "broken" examples—and Ensemble Learning (EL)—combining multiple models to make a joint decision. However, simply combining AT and EL is surprisingly difficult. Just stacking adversarially trained models often yields diminishing returns or even hurts the ensemble's overall robustness. This happens because models struggle to balance learning from clean vs. adversarial data, often becoming "over-confident" in ways that waste their limited learning capacity.
The AUTE Framework
To address these challenges, we introduce AUTE (Alignment and Unlearning for Training Ensembles), which consists of two novel components designed to make ensemble members work better together:
- Peer-Alignment (PA): Instead of training models in isolation, AUTE uses a round-robin approach. It partitions the ensemble into a "single member" (the student) and a "peer model" (a frozen ensemble of all other members). The student is encouraged to align its internal features with the peer model’s robust features. Crucially, this alignment is selective: the student only aligns with the peer when it is less confident than the peer, preventing the "blind leading the blind" if the peer model happens to be wrong.
- Self-Unlearning (SU): Standard AT forces a model to try and "memorize" every adversarial example. AUTE argues this is a mistake. It identifies data points where the model is already over-confident (high "margin distance") and applies a negative weight to the loss function. This forces the model to "forget" these easy or over-fitted points, freeing up its capacity to learn more complex, robust features that improve generalization.
Experimental Evaluation
We evaluated AUTE using several standard computer vision benchmarks, including MNIST, CIFAR-10, CIFAR-100, and Tiny-ImageNet, across various architectures like ResNet and WideResNet. AUTE consistently outperformed state-of-the-art (SOTA) baselines like TRADES, MART, and AROW.
Key Results
- Robustness Gains: On CIFAR-10 with a 5-member ResNet-20 ensemble, AUTE achieved 55.5% average robustness against various attacks, significantly higher than the runner-up AROW (52.3%).
- Clean Accuracy: Unlike many defense methods that significantly degrade performance on normal data, AUTE maintained high clean accuracy (e.g., 78.0% on CIFAR-10).
- Scalability: Many ensemble methods plateau quickly. AUTE shows a clear upward trend in robustness as the ensemble size grows from 3 to 8 members, suggesting it makes better use of additional parameters.
- Standard Training: Even when trained only on natural data, AUTE's framework boosted "black-box" robustness more effectively than specialized ensemble diversity methods like DVERGE or TRS.
Conclusion
AUTE demonstrates that "forgetting" certain over-fitted data points and aligning with collective peer knowledge can significantly bolster defenses against adversarial attacks. By managing model capacity more effectively, AUTE provides a scalable and high-performance path forward for robust ensemble learning.
Summary updated