Towards Adversarial Robustness via Debiased High-Confidence Logit Alignment

Kejia Zhang, Juanjuan Weng, Shaozi Li, Zhiming Luo

TL;DR

Adversarial training methods that use high-confidence "inverse" adversarial examples inadvertently cause models to overfit to irrelevant background features. To fix this, the authors propose Debiased High-Confidence Adversarial Training (DHAT), which strips out background bias from high-confidence logits and enforces foreground feature focus. This approach achieves state-of-the-art adversarial robustness and significantly reduces the robust generalization gap across major benchmarks like CIFAR and ImageNet.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) are notoriously vulnerable to adversarial attacks—tiny, imperceptible perturbations designed to fool them. To defend against this, researchers use Adversarial Training (AT), which injects adversarial examples into the training loop.

A recent, clever twist in AT involves using inverse adversarial attacks. Instead of making a model fail, inverse attacks find perturbations that make the model more confident in its correct predictions. By aligning standard adversarial examples with these high-confidence examples, models learn more robust decision boundaries.

However, this paper uncovers a critical flaw in this strategy: spurious correlation bias. Under inverse attacks, the model achieves high confidence not by learning better core features of an object, but by over-activating on the background context (e.g., identifying a sheep by recognizing the grass it stands on rather than the animal itself). This causes severe "robust overfitting," where the model performs exceptionally well on the training set but fails to generalize to new adversarial examples because it is looking at the wrong parts of the image.

Figure 1. Inverse adversarial attack exploits spurious correlations via attention misallocation to non-causal background features. The negative ϵ denotes inverse adversarial perturbation strength. (a) Grad-CAM [34] visualizations show that adversarial attacks dis- rupt attention and cause errors, while inverse adversarial attacks amplify activation and improve accuracy but expose a critical flaw: attention shifts from foreground to irrelevant background regions. (b) Quantitative comparison of IoU(%) between atten- tion regions and the true foreground and background regions using UIAT [9] and AT. Increasing inverse attack strength −ϵ demon- strates UIAT’s disproportionate focus on non-causal background features, exposing reliance on spurious correlations [3, 30, 35].

Methods

To eliminate this background bias, the authors introduce Debiased High-Confidence Adversarial Training (DHAT). DHAT modifies the standard min-max adversarial training objective using two novel components:

  1. Debiased High-Confidence Logit Regularization (DHLR): DHAT uses a saliency method (like Grad-CAM) to generate an attention map of a natural image. It then masks out the foreground on the corresponding high-confidence inverse adversarial example to isolate the background features. By passing this background-only image through the network, it calculates "background logits." DHLR subtracts these background logits from the total high-confidence logits to create a purified, debiased target. The standard adversarial logits are then aligned with this debiased target using Kullback–Leibler (KL) divergence.

  2. Foreground Logit Orthogonal Enhancement (FLOE): To actively discourage the model from relying on background features during training, FLOE minimizes the mathematical correlation between the high-confidence logits and the background logits in affine space. It forces the projection of the full logits onto the background logits to be as small as possible, ensuring predictions are driven by foreground information.

Figure 2. Overview of Debiased High-Confidence Logits Regu- larization (DHLR). The method first quantifies the degree of bi- ased activation towards background features and recalibrates bi- ased logits, then aligns the logits of adversarial examples with de- biased high-confidence logits.

Datasets

The authors evaluated their approach using several standard computer vision architectures (including WRN28-10, ResNet-18, ResNet-50, VGG16, and Inception-V3) across three primary datasets:

  • CIFAR-10
  • CIFAR-100
  • ImageNet-1K
  • Imagenette (a smaller subset of ImageNet used for specific attention map evaluations)

Additionally, they tested DHAT's performance when augmented with 50K synthetic images generated by a Diffusion Denoising Probabilistic Model (DDPM).

Key Insights and Ideas

  • The Illusion of Confidence: High confidence does not equal high-quality feature learning. Inverse adversarial examples amplify activations, but they systematically shift the model's attention away from causal foreground features toward non-causal background contexts.
  • The Root of Robust Overfitting: Overfitting to background features is a primary driver of the "Robust Gap"—the performance drop between training and testing sets under attack.
  • Logit Arithmetic for Debiasing: Simply subtracting background-derived logits from full logits is an elegant and effective way to mathematically "strip" spurious context bias before distribution alignment.

Key Results

DHAT consistently outperforms state-of-the-art adversarial defense methods across multiple architectures and benchmarks:

  • Drastic Reduction in Robust Overfitting: On CIFAR-10, DHAT reduced the robust generalization gap to just 3.51%, compared to 7.92% for the inverse-attack baseline (UIAT) and 9.52% for MART.
  • Top-Tier Robustness: Under the rigorous AutoAttack (AA) benchmark on CIFAR-10, DHAT achieved 53.10% accuracy. When combined with Class-wise Calibrated Fair AT (CFA), it reached 54.05% accuracy, outperforming all baselines.
  • Scalability to ImageNet: On ImageNet-1K, DHAT achieved a test accuracy of 41.70% under AutoAttack, outperforming UIAT (40.18%) while slashing the robust gap from 14.68% down to 9.53%.
Figure 3. Comparisons of robustness (%) under varying ϵ adver- sarial attacks using ResNet-18 on CIFAR-10.

Limitations and Future Work

  • Computational Overhead: Generating attention maps (via Grad-CAM) and running an extra forward pass for background features introduces a minor computational cost per epoch compared to standard AT, though it remains faster than some complex baselines like MART or FSR.
  • Advanced Saliency Costs: While DHAT is compatible with more precise segmentation models like the Segment Anything Model (SAM), using them significantly increases training time, creating a trade-off between optimal performance and computational efficiency.
  • Hyperparameter Tuning: The framework introduces three hyperparameters ($\lambda_1, \lambda_2$, and the attention threshold $\omega$) that require careful calibration depending on dataset resolution and complexity.

Summary updated

Abstract

Despite the remarkable progress of deep neural networks (DNNs) in various visual tasks, their vulnerability to adversarial examples raises significant security concerns. Recent adversarial training methods leverage inverse adversarial attacks to generate high-confidence examples, aiming to align adversarial distributions with high-confidence class regions. However, our investigation reveals that under inverse adversarial attacks, high-confidence outputs are influenced by biased feature activations, causing models to rely on background features that lack a causal relationship with the labels. This spurious correlation bias leads to overfitting irrelevant background features during adversarial training, thereby degrading the model's robust performance and generalization capabilities. To address this issue, we propose Debiased High-Confidence Adversarial Training (DHAT), a novel approach that aligns adversarial logits with debiased high-confidence logits and restores proper attention by enhancing foreground logit orthogonality. Extensive experiments demonstrate that DHAT achieves state-of-the-art robustness on both CIFAR and ImageNet-1K benchmarks, while significantly improving generalization by mitigating the feature bias inherent in inverse adversarial training approaches. Code is available at https://github.com/KejiaZhang-Robust/DHAT.
BibTeX
@inproceedings{zhang2025towards,
  author = {Kejia Zhang and Juanjuan Weng and Shaozi Li and Zhiming Luo},
  title = {{Towards Adversarial Robustness via Debiased High-Confidence Logit Alignment}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
  url = {https://openreview.net/forum?id=s0kG2Bq4FN},
}
Figure