Most research in adversarial training (AT) focuses on better loss functions and optimizers, but the network architecture itself is a critical, often overlooked factor. Deep learning models are notoriously vulnerable to adversarial attacks—small, intentional perturbations that can trick a model into making confident, incorrect predictions. To counter this, researchers use Adversarial Training (AT), where models are trained on these perturbed examples.
However, the community has largely relied on classic architectures (like ResNet or WideResNet) designed for standard training. This leads to a "misalignment phenomenon": a model that is highly accurate on clean data isn't necessarily the most robust under attack. While Neural Architecture Search (NAS) could find better structures, performing NAS during the computationally expensive AT process is practically infeasible.
To address this, researchers introduce DS-Net (Diverse-Structured Network), which uses learnable attention weights to combine diverse "atomic" building blocks, achieving superior adversarial robustness and faster convergence than traditional human-designed or searched architectures.
The DS-Net Architecture
DS-Net shifts the focus from searching for a single "best" architecture to weighting a diverse set of proven structures concurrently during training.
- Robust Modules: Instead of a single sequence of layers, DS-Net consists of sequential modules. Each module contains parallel "atomic blocks"—time-tested structures like Residual blocks, Dense blocks, and Adaptive Runge-Kutta (Ada-RK) blocks.
- Learnable Attention Weights: For each module, the model learns a set of attention weights. The output of a module is the weighted sum of the outputs from all its internal atomic blocks.
- Concurrent Optimization: Unlike traditional NAS, which often involves a two-stage "search then retrain" process, DS-Net optimizes the block parameters and the attention weights simultaneously during the standard AT loop.
Key Insights: Why Diversity Works
- The Power of Diversity: The core philosophy is that "weighting" is better than "picking." By keeping a diversity of structures available throughout training, the model can exploit the strengths of different blocks (e.g., Dense blocks for feature reuse, Residual blocks for gradient flow) at different depths.
- Smoothing the Landscape: Theoretically, DS-Net has a smaller Lipschitz constant than standard architectures. This means the model's output changes less drastically in response to small input changes, which is the mathematical definition of robustness.
- Better Convergence: The diverse structure leads to lower gradient variance during training. This makes the optimization process more stable and allows the model to converge to a robust solution faster than traditional networks.
- Location Sensitivity: Visualizing the learned weights reveals that the model naturally prefers different blocks at different stages. For instance, it tends to favor Residual blocks in later layers to provide "cleaner" features for final classification.
Performance and Results
Evaluated on standard image classification benchmarks (CIFAR-10 and SVHN), DS-Net consistently outperformed state-of-the-art baselines across multiple attack types (PGD, AutoAttack, C&W) and training styles (Standard AT, TRADES, MART).
- Efficiency: A "small" DS-Net with only 20.78M parameters achieved 54.14% accuracy under PGD-20 attack on CIFAR-10, outperforming a much larger WideResNet-34-10 (46.16M parameters) which scored 51.95%.
- Peak Robustness: The larger DS-Net-6-softmax achieved the best overall results, reaching 51.48% accuracy against the rigorous AutoAttack benchmark, a significant improvement over standard ResNet and WideResNet models.
- Generalization: Unlike some robust models that sacrifice too much clean accuracy, DS-Net maintains high performance on natural, unperturbed data (e.g., 86.76% on CIFAR-10).
Limitations and Future Work
While highly effective, DS-Net has a few limitations. The current "atomic blocks" are manually selected based on existing literature; a more principled approach to selecting these initial candidates could further improve results. Interestingly, the authors found that "pruning" the less-weighted blocks after training (a common NAS tactic) actually hurts performance, suggesting the blocks are highly co-adapted and diversity itself is essential. Finally, while successful on CIFAR and SVHN, future work is needed to see how these diverse-structured networks scale to much larger, high-resolution datasets like ImageNet.
Summary updated