Introduction
Deep neural networks are notoriously vulnerable to adversarial attacks—small, carefully crafted perturbations that are invisible to humans but cause models to make significant errors. While adversarial training (AT) is a standard defense in image classification, it has seen limited success in semantic segmentation. In segmentation tasks, every pixel represents a classification sub-task, and standard AT often leads to a "robustness-accuracy trade-off," where defending against noise significantly degrades performance on clean images.
This paper introduces Dynamic Divide-and-Conquer Adversarial Training (DDC-AT), a novel strategy that enhances robustness without sacrificing clean-image accuracy or increasing inference-time computational costs.
Methodology: The Divide-and-Conquer Strategy
The core of DDC-AT is the recognition of pixel heterogeneity: not all pixels are equally vulnerable. Some are "safe" (far from decision boundaries), while others are "sensitive" (near class borders). DDC-AT uses three specialized branches sharing a common backbone during training:
- Main Branch ($f_n$): Handles adversarial pixels and robust clean pixels. This is the only branch kept for the final deployed model.
- Auxiliary Branch ($f_a$): Specifically handles sensitive clean pixels that are likely to be flipped by small noise.
- Mask Branch ($f_m$): A lightweight component that dynamically predicts which pixels belong in which branch based on their "boundary property."
As training progresses, pixels that were initially sensitive gradually become more robust. The Mask Branch learns to move these pixels into the Main Branch. By the end of training, the Main Branch effectively handles both clean and adversarial pixels. Crucially, the auxiliary and mask branches are discarded after training, meaning the final model maintains the original architecture's speed and size.
Experiments and Results
The researchers evaluated DDC-AT on two major benchmarks: PASCAL VOC 2012 (object-centric) and Cityscapes (urban street scenes), using popular architectures like PSPNet and DeepLabv3.
Robustness vs. Accuracy
DDC-AT demonstrates superior performance in both robustness and clean-image accuracy compared to Standard Adversarial Training (SAT). On PASCAL VOC with PSPNet under a strong BIM attack:
- No Defense: 5.4% mIoU
- Standard AT: 27.9% mIoU
- DDC-AT: 33.8% mIoU
While SAT dropped clean accuracy from 76.9% to 74.3%, DDC-AT maintained a much higher 76.0%. This suggests that the divide-and-conquer approach successfully mitigates the typical trade-off seen in adversarial defenses.
Optimization Stability
Beyond raw metrics, DDC-AT showed significantly lower standard deviation across training runs compared to SAT. By separating sensitive pixels into an auxiliary branch, the Main Branch can focus on a more stable distribution, leading to a more reliable optimization process.
Key Insights
- Curriculum-like Learning: The dynamic movement of pixels from the auxiliary to the main branch acts as a natural curriculum, allowing the model to master easier pixels before tackling those near decision boundaries.
- Zero-Cost Robustness: DDC-AT proves that robustness can be "baked" into the weights of a standard architecture during training, requiring no extra layers or multiple passes at runtime.
Limitations and Future Work
While DDC-AT is highly effective, it increases training complexity and time due to the need for generating adversarial examples in every batch. Future research could explore the generalization of this branching strategy to other dense prediction tasks, such as instance segmentation or depth estimation, and investigate defenses against adaptive attackers who might target the Mask Branch during training.
Summary updated