Dynamic Divide-and-Conquer Adversarial Training for Robust Semantic Segmentation

Xiaogang Xu, Hengshuang Zhao, Jiaya Jia

Introduction

Deep neural networks are notoriously vulnerable to adversarial attacks—small, carefully crafted perturbations that are invisible to humans but cause models to make significant errors. While adversarial training (AT) is a standard defense in image classification, it has seen limited success in semantic segmentation. In segmentation tasks, every pixel represents a classification sub-task, and standard AT often leads to a "robustness-accuracy trade-off," where defending against noise significantly degrades performance on clean images.

This paper introduces Dynamic Divide-and-Conquer Adversarial Training (DDC-AT), a novel strategy that enhances robustness without sacrificing clean-image accuracy or increasing inference-time computational costs.

Methodology: The Divide-and-Conquer Strategy

The core of DDC-AT is the recognition of pixel heterogeneity: not all pixels are equally vulnerable. Some are "safe" (far from decision boundaries), while others are "sensitive" (near class borders). DDC-AT uses three specialized branches sharing a common backbone during training:

  1. Main Branch ($f_n$): Handles adversarial pixels and robust clean pixels. This is the only branch kept for the final deployed model.
  2. Auxiliary Branch ($f_a$): Specifically handles sensitive clean pixels that are likely to be flipped by small noise.
  3. Mask Branch ($f_m$): A lightweight component that dynamically predicts which pixels belong in which branch based on their "boundary property."

As training progresses, pixels that were initially sensitive gradually become more robust. The Mask Branch learns to move these pixels into the Main Branch. By the end of training, the Main Branch effectively handles both clean and adversarial pixels. Crucially, the auxiliary and mask branches are discarded after training, meaning the final model maintains the original architecture's speed and size.

Experiments and Results

The researchers evaluated DDC-AT on two major benchmarks: PASCAL VOC 2012 (object-centric) and Cityscapes (urban street scenes), using popular architectures like PSPNet and DeepLabv3.

Robustness vs. Accuracy

DDC-AT demonstrates superior performance in both robustness and clean-image accuracy compared to Standard Adversarial Training (SAT). On PASCAL VOC with PSPNet under a strong BIM attack:

  • No Defense: 5.4% mIoU
  • Standard AT: 27.9% mIoU
  • DDC-AT: 33.8% mIoU

While SAT dropped clean accuracy from 76.9% to 74.3%, DDC-AT maintained a much higher 76.0%. This suggests that the divide-and-conquer approach successfully mitigates the typical trade-off seen in adversarial defenses.

Qualitative comparison of segmentation results under adversarial attack. The columns likely represent the original image, standard adversarial training, and the proposed DDC-AT results, showing clearer boundaries and fewer artifacts in the latter.

Optimization Stability

Beyond raw metrics, DDC-AT showed significantly lower standard deviation across training runs compared to SAT. By separating sensitive pixels into an auxiliary branch, the Main Branch can focus on a more stable distribution, leading to a more reliable optimization process.

A detailed segmentation mask from the Cityscapes dataset, illustrating the complexity of urban scene parsing where many small objects and boundaries create high sensitivity to adversarial noise.

Key Insights

  • Curriculum-like Learning: The dynamic movement of pixels from the auxiliary to the main branch acts as a natural curriculum, allowing the model to master easier pixels before tackling those near decision boundaries.
  • Zero-Cost Robustness: DDC-AT proves that robustness can be "baked" into the weights of a standard architecture during training, requiring no extra layers or multiple passes at runtime.

Limitations and Future Work

While DDC-AT is highly effective, it increases training complexity and time due to the need for generating adversarial examples in every batch. Future research could explore the generalization of this branching strategy to other dense prediction tasks, such as instance segmentation or depth estimation, and investigate defenses against adaptive attackers who might target the Mask Branch during training.

Summary updated

Abstract

Adversarial training is promising for improving robustness of deep neural networks towards adversarial perturbations, especially on the classification task. The effect of this type of training on semantic segmentation, contrarily, just commences. We make the initial attempt to explore the defense strategy on semantic segmentation by formulating a general adversarial training procedure that can per-form decently on both adversarial and clean samples. We propose a dynamic divide-and-conquer adversarial training (DDC-AT) strategy to enhance the defense effect, by set-ting additional branches in the target model during training, and dealing with pixels with diverse properties to-wards adversarial perturbation. Our dynamical division mechanism divides pixels into multiple branches automatically. Note all these additional branches can be abandoned during inference and thus leave no extra parameter and computation cost. Extensive experiments with various segmentation models are conducted on PASCAL VOC 2012 and Cityscapes datasets, in which DDC-AT yields satisfying performance under both white- and black-box at-tack. The code is available at https://github.com/dvlab-research/Robust-Semantic-Segmentation.
BibTeX
@inproceedings{xu2020dynamic,
  author = {Xiaogang Xu and Hengshuang Zhao and Jiaya Jia},
  title = {{Dynamic Divide-and-Conquer Adversarial Training for Robust Semantic Segmentation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2020},
  doi = {10.1109/iccv48922.2021.00739},
}
Figure