Introduction
Deep Neural Networks (DNNs) are highly susceptible to adversarial attacks, where small, calculated perturbations to input data lead to incorrect classifications. While Adversarial Training (AT) provides a robust defense, it typically forces a trade-off: increasing robustness often degrades accuracy on clean, unperturbed images. Traditional AT models are static, fixed at a single point on this trade-off curve. Adapting to varying environmental conditions (e.g., switching from high accuracy in clear weather to high robustness in stormy conditions) usually requires deploying multiple models, which is impractical for resource-constrained edge devices.
The FLOAT Framework
To address these challenges, the authors propose FLOAT (Fast Learnable Once-for-All Adversarial Training). FLOAT enables a single neural network to dynamically adjust its trade-off between clean-image accuracy and adversarial robustness at inference time without the need for additional conditioning layers or significant parameter overhead.
Methodology
FLOAT introduces three core architectural and training innovations:
- Weight-Conditioned Learning: Unlike previous "Once-for-All" (OAT) methods that use Feature-wise Linear Modulation (FiLM) layers to adjust activations, FLOAT modulates the network's weight tensors directly. It adds a learnable noise tensor ($\alpha' * \eta^I$) to the original weights ($\theta'$), controlled by a switch ($\lambda$). This allows the model to capture distinct feature distributions for clean and adversarial inputs within the same weight space.
- Dual Batch Normalization (BN): Clean and adversarial images exhibit different statistical properties. FLOAT employs two independent BN branches ($BN_C$ for clean, $BN_A$ for adversarial) to track these distributions separately, switching between them based on the input type.
- Inference-Time Tuning: At test time, users can provide a continuous hyperparameter $\lambda_n \in [0, 1]$. This parameter scales the weight modulation, allowing the model to smoothly transition between its "clean" and "robust" operational modes.
Experimental Evaluation
The researchers evaluated FLOAT and its sparse variant, FLOATS, across several benchmarks, including CIFAR-10/100, SVHN, STL10, and a subset of ImageNet. They tested these on popular architectures like ResNet34 and WideResNet (WRN16-8, WRN40-2).
Key Results
FLOAT demonstrates significant improvements over existing state-of-the-art (SOTA) OAT methods in terms of performance, efficiency, and speed:
- Performance Gains: FLOAT achieved up to a ~6% increase in clean accuracy and a ~10% increase in robust accuracy compared to previous methods.
- Efficiency: The framework uses up to 1.47× fewer parameters than FiLM-based OAT. The FLOATS slim variant further improves parameter efficiency by up to 2.69×.
- Speed: Training time is reduced by up to 1.43× because FLOAT does not require training on a dense grid of intermediate trade-off values. Furthermore, the absence of extra layers results in up to 1.66× faster inference latency on specialized hardware.
Conclusion
FLOAT provides a hardware-friendly, efficient solution for deploying tunable robust models on edge devices. By shifting the conditioning mechanism from activations to weights and utilizing dual batch normalization, FLOAT achieves superior accuracy-robustness trade-offs with lower computational overhead. Future work may explore automated selection of the tuning parameter $\lambda_n$ based on real-time environmental context or model confidence.
Summary updated