Abstract
For years, WideResNets (WRNs) were considered the undisputed champions of adversarial robustness on small-scale datasets like CIFAR-10. This work shatters that consensus, demonstrating that Vision Transformers (ViTs)—when fueled by diffusion-generated synthetic data and optimized with the Lion optimizer—can significantly outperform WRNs. We set new state-of-the-art benchmarks for both clean and robust accuracy, achieving 74.97% robust accuracy on CIFAR-10, a 3.9% improvement over previous bests.
Introduction
In deep learning, "adversarial robustness" refers to a model's resilience against human-imperceptible perturbations designed to induce misclassification. While Vision Transformers (ViTs) have dominated large-scale computer vision tasks, they were long deemed unsuitable for small-scale adversarial training due to their high data requirements and tendency to overfit on limited datasets like CIFAR-10 or CIFAR-100. Consequently, WideResNets (WRNs) remained the gold standard for these benchmarks. This research challenges the status quo, investigating whether ViTs can surpass WRNs in adversarial robustness on small-scale datasets.
Methodology
The breakthrough is achieved by integrating three critical components into the training pipeline:
- Massive Synthetic Data Generation: To address the ViT's data hunger, the authors utilized the Elucidating Diffusion Model (EDM) to generate 50 million synthetic images. This massive augmentation helps the model learn generalized features and mitigates overfitting.
- TRADES Framework: The TRADES (Theoretically Principled Trade-off between Robustness and Accuracy) loss function was employed to explicitly balance clean performance and adversarial resilience.
- Lion Optimizer: Standard SGD, while effective for CNNs, proved inadequate for robust ViTs. The Lion optimizer provided the necessary stability and performance gains required for this architecture.
- Extended Training Schedules: Training was scaled up to 10,000 epochs to fully optimize the larger ViT-L models.
Experimental Results
The models were evaluated using AutoAttack (AA) on the RobustBench leaderboard, focusing on CIFAR-10 and CIFAR-100 benchmarks.
The results demonstrate a significant leap over previous state-of-the-art (SOTA) models:
- CIFAR-10 ($\ell_\infty, \epsilon = 8/255$): Achieved 74.97% robust accuracy (+3.9% over previous SOTA).
- CIFAR-100 ($\ell_\infty, \epsilon = 8/255$): Achieved 44.07% robust accuracy (+1.4% over previous SOTA).
- Clean Accuracy: The ViT-L/2 model reached 95.76% clean accuracy on CIFAR-10, becoming the first model to simultaneously exceed 95.5% clean and 74.5% robust accuracy.
Key Insights
- Holistic Feature Representation: EigenCAM visualizations reveal that ViTs focus on the entire semantic object, whereas WRNs often fixate on local patches. This holistic understanding likely contributes to their superior robustness against local noise.
- Architectural Efficiency: The ViT-B/2 model outperformed the top-tier WRN-70-16 despite having 3x fewer parameters, suggesting that Transformers are fundamentally more efficient at capturing robust features when provided with sufficient data.
- Data Mix Ratios: A 9:1 ratio of synthetic to real data was found to be optimal, confirming that ViTs thrive on massive, diverse datasets.
Conclusion and Future Work
This work establishes ViTs as the new leaders in adversarial robustness for small-scale datasets. While the computational cost of training (thousands of epochs on TPU clusters) is high, the performance gains are undeniable. Future research may explore more efficient Transformer variants, such as Swin Transformers, and evaluate generalization across a broader range of datasets.
Summary updated