Improving White-box Robustness of Pre-processing Defenses via Joint Adversarial Training

Dawei Zhou, Nannan Wang, Xinbo Gao, Bo Han, Jun Yu, Xiaoyu Wang, Tongliang Liu

Abstract

Deep neural networks (DNNs) are vulnerable to adversarial noise. A range of adversarial defense techniques have been proposed to mitigate the interference of adversarial noise, among which the input pre-processing methods are scalable and show great potential to safeguard DNNs. However, pre-processing methods may suffer from the robustness degradation effect, in which the defense reduces rather than improving the adversarial robustness of a target model in a white-box setting. A potential cause of this negative effect is that adversarial training examples are static and independent to the pre-processing model. To solve this problem, we investigate the influence of full adversarial examples which are crafted against the full model, and find they indeed have a positive impact on the robustness of defenses. Furthermore, we find that simply changing the adversarial training examples in pre-processing methods does not completely alleviate the robustness degradation effect. This is due to the adversarial risk of the pre-processed model being neglected, which is another cause of the robustness degradation effect. Motivated by above analyses, we propose a method called Joint Adversarial Training based Pre-processing (JATP) defense. Specifically, we formulate a feature similarity based adversarial risk for the pre-processing model by using full adversarial examples found in a feature space. Unlike standard adversarial training, we only update the pre-processing model, which prompts us to introduce a pixel-wise loss to improve its cross-model transferability. We then conduct a joint adversarial training on the pre-processing model to minimize this overall risk. Empirical results show that our method could effectively mitigate the robustness degradation effect across different target models in comparison to previous state-of-the-art approaches.

The Challenge: The Robustness Degradation Effect

Input pre-processing defenses, such as image denoisers, are designed to "clean" images before they reach a target model. However, in a white-box setting where an attacker is aware of the defense, these methods often suffer from the robustness degradation effect. As illustrated in the experimental results, standard defenses can paradoxically make a system less robust than having no defense at all.

This happens because most denoisers are trained "obliviously"—they practice on noise meant for the target model alone, failing to account for an adaptive attacker who targets the combined pipeline of the denoiser and the model. As shown in the comparison of standard and TRADES models, the white-box robustness of defended models (APE-G, HGD, NRP) is frequently lower than the baseline with no defense ("None").

Robustness degradation effect in standard and TRADES models, and the initial benefit of 'full' adversarial training.

Joint Adversarial Training based Pre-processing (JATP)

To solve this, we propose Joint Adversarial Training based Pre-processing (JATP). This method shifts the training paradigm for pre-processing defenses in three key ways:

  1. Full Adversarial Examples: Instead of training on static noise, JATP crafts "full" adversarial examples designed to fool the entire pipeline (the denoiser and the target model together).
  2. Feature Similarity Metric (FSM): JATP minimizes the distance between natural and adversarial images within the feature space (internal layers) of the model. This prevents the denoiser from being misled into producing features that cause misclassification.
  3. Pixel-wise Loss ($L_{pix}$): To ensure the denoiser remains effective across different target models (cross-model transferability), JATP includes a pixel-wise loss that keeps the cleaned image visually similar to the original.

Key Results and Visual Analysis

Empirical evaluations on CIFAR-10 and SVHN demonstrate that JATP effectively mitigates the robustness degradation effect. By training with "full" adversarial examples, the defense maintains robustness levels comparable to or better than the original model, significantly outperforming "oblivious" training methods.

White-box robustness comparison of JATP-style 'full' training vs oblivious training and visual analysis of pre-processed images.

Visual analysis reveals that while JATP-processed images may have higher pixel-level differences ($L_1$ distance) from natural images compared to raw adversarial images, they are far more effective at neutralizing perturbations in the feature space. This ensures that the classification model receives "clean" features, even when the input pixels have been significantly modified to counteract noise.

Conclusion

JATP addresses the fundamental flaws of oblivious pre-processing by treating the defense as a first-class citizen in the adversarial training process. By optimizing for both feature similarity and pixel-wise fidelity, JATP provides a scalable and transferable solution that genuinely enhances the white-box robustness of deep learning systems.

Summary updated

Abstract

Deep neural networks (DNNs) are vulnerable to adversarial noise. A range of adversarial defense techniques have been proposed to mitigate the interference of adversarial noise, among which the input pre-processing methods are scalable and show great potential to safeguard DNNs. However, pre-processing methods may suffer from the robustness degradation effect, in which the defense reduces rather than improving the adversarial robustness of a target model in a white-box setting. A potential cause of this negative effect is that adversarial training examples are static and independent to the pre-processing model. To solve this problem, we investigate the influence of full adversarial examples which are crafted against the full model, and find they indeed have a positive impact on the robustness of defenses. Furthermore, we find that simply changing the adversarial training examples in pre-processing methods does not completely alleviate the robustness degradation effect. This is due to the adversarial risk of the pre-processed model being neglected, which is another cause of the robustness degradation effect. Motivated by above analyses, we propose a method called Joint Adversarial Training based Pre-processing (JATP) defense. Specifically, we formulate a feature similarity based adversarial risk for the pre-processing model by using full adversarial examples found in a feature space. Unlike standard adversarial training, we only update the pre-processing model, which prompts us to introduce a pixel-wise loss to improve its cross-model transferability. We then conduct a joint adversarial training on the pre-processing model to minimize this overall risk. Empirical results show that our method could effectively mitigate the robustness degradation effect across different target models in comparison to previous state-of-the-art approaches.
BibTeX
@misc{zhou2021improving,
  author = {Dawei Zhou and Nannan Wang and Xinbo Gao and Bo Han and Jun Yu and Xiaoyu Wang and Tongliang Liu},
  title = {{Improving White-box Robustness of Pre-processing Defenses via Joint Adversarial Training}},
  year = {2021},
  eprint = {2106.05453},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.05453},
}
Figure