3SAT: A Simple Self-Supervised Adversarial Training Framework

Jiang Fang, Haonan He, Jiyan Sun, Jiadong Fu, Zhaorui Guo, Yinlong Liu, Wei MaAAAI 2025 · DOI

TL;DR

This paper introduces 3SAT, a novel self-supervised adversarial training framework that significantly boosts model robustness without sacrificing accuracy. By training directly on original, unaugmented samples and employing a dynamic objective scheduling, 3SAT achieves state-of-the-art results, even surpassing supervised adversarial training methods for the first time.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) are notoriously vulnerable to adversarial attacks, where imperceptible perturbations can lead to incorrect predictions. While adversarial training (AT) is a common defense, its application to self-supervised learning (SSL) models, which are increasingly matching or exceeding supervised models in performance, has lagged behind. Current self-supervised adversarial training (self-AT) methods struggle with a complex joint optimization problem and are negatively impacted by the data augmentation techniques crucial for SSL. This limits the secure deployment of powerful SSL models in critical applications.

Methods

The core innovation of 3SAT lies in its simplified approach to self-AT. Instead of using augmented samples for adversarial training, 3SAT directly employs original, unaugmented samples and their corresponding adversarial perturbations. This simplifies the adversarial training subproblem and eliminates the negative impact of data augmentation on robustness.

To address the potential issue of training collapse due to the similarity of original and adversarial samples, 3SAT introduces a Dynamic Training Objective Scheduling (DTOS) strategy. This strategy first allows the self-supervised learning component to "warm up" and learn good representations before gradually increasing the strength of adversarial training. This ensures robust representations are learned before focusing on adversarial defense.

Furthermore, 3SAT abandons the Dual Batch Normalization (BN) technique commonly used in prior self-AT methods. This simplification reduces model parameters and computational overhead.

Datasets

The paper evaluates 3SAT on the following datasets:

  • CIFAR-10
  • CIFAR-100
  • STL-10

Key Insights and Ideas

  • Original Samples are Key for Robustness: The paper demonstrates that using original, unaugmented samples for adversarial training is crucial for achieving high robustness without compromising standard accuracy. This challenges the common practice of relying on augmented data in self-AT.
  • Dynamic Scheduling Prevents Collapse: The DTOS strategy is vital for stabilizing training. By prioritizing representation learning before robust training, it prevents the model from getting stuck in trivial solutions.
  • Simplicity Enhances Efficiency: Removing Dual BN and using original samples leads to a simpler, more computationally efficient framework that halves training time compared to previous SOTA methods.
  • Self-AT Can Surpass Supervised AT: The most striking insight is that 3SAT, a self-AT method, achieves superior robust accuracy compared to state-of-the-art supervised AT methods, suggesting a paradigm shift in how we approach robust model development.

Key Results

  • Robust Accuracy Improvement: On CIFAR-10, 3SAT improves the robust accuracy of the SOTA self-AT method by 16.19% and standard accuracy by 11.41% under Auto-Attack.
  • Outperforming Supervised AT: 3SAT surpasses the SOTA supervised AT method by a significant margin of 11.25% in robust accuracy.
  • Training Speed: 3SAT reduces the training time by half compared to previous SOTA self-AT methods.
  • Label Efficiency: In semi-supervised settings, 3SAT demonstrates strong performance even with as little as 1% of labels, outperforming SOTA self-AT methods that use all labels.
  • Visualizations: UMAP visualizations show that 3SAT learns representations with clearer class boundaries, indicating better discriminative power. Loss landscape visualizations reveal a flatter landscape for 3SAT, correlating with better robust generalization.

Limitations and Future Work

  • Warm-up Sensitivity: While 3SAT is not highly sensitive to the exact warm-up duration, the paper notes that for complex datasets like CIFAR-100, a sufficient warm-up phase (e.g., 200 epochs) is crucial for optimal standard and robust accuracy.
  • Future Work: The paper implicitly suggests that the success of 3SAT opens avenues for further exploring the decoupling of SSL and AT objectives and investigating the benefits of using original samples in adversarial training across various domains and architectures. The authors also highlight that 3SAT's superior performance indicates self-AT might be a more promising direction for achieving robust models.

Summary updated

BibTeX
@inproceedings{fang2025sat,
  author = {Jiang Fang and Haonan He and Jiyan Sun and Jiadong Fu and Zhaorui Guo and Yinlong Liu and Wei Ma},
  title = {{3SAT: A Simple Self-Supervised Adversarial Training Framework}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i16.33815},
}
Figure