Abstract
Adversarial attacks on image segmentation models have traditionally relied on strategies adapted from image classification, often failing to account for the unique pixel-wise nature of segmentation tasks. This paper introduces a novel attack framework that repurposes the certified radius—a metric typically used to measure model robustness—as a guide for identifying and prioritizing vulnerable pixels. By focusing adversarial perturbations on these "low-hanging fruit," the proposed framework achieves significant performance gains in both white-box (CR-PGD) and black-box (CR-PBGD) settings. Experimental results across Pascal VOC, Cityscapes, and ADE20K benchmarks demonstrate that this approach outperforms standard methods while maintaining computational efficiency.
1. Introduction
Image segmentation is a cornerstone of safety-critical computer vision applications, including autonomous driving and medical imaging. Unlike image classification, which assigns a single label to an entire image, segmentation requires a prediction for every individual pixel. While it is well-established that these models are susceptible to adversarial perturbations, existing attack methodologies often distribute their perturbation budgets inefficiently across the image.
This research addresses a fundamental question: How can an attacker characterize the inherent vulnerability of individual pixels to craft a more surgical and effective attack? We propose that the certified radius, derived from randomized smoothing, provides an ideal roadmap for such an endeavor. By weaponizing a defense metric, we demonstrate that pixels with smaller certified radii are significantly easier to flip, allowing for a more strategic allocation of adversarial noise.
2. Methodology
2.1 The Certified Radius as a Vulnerability Map
The core innovation of this framework is the use of Randomized Smoothing to derive a pixel-wise certified radius. In a defensive context, a larger certified radius indicates that a pixel's prediction is robust against a certain magnitude of noise. Conversely, from an adversarial perspective, a small certified radius identifies a pixel as highly vulnerable. This inverse relationship allows us to create a "vulnerability map" that guides the attack process.
2.2 White-Box Attack: CR-PGD
In the white-box setting, we introduce CR-PGD, which utilizes a Certified Radius-Guided Loss. Rather than treating all pixels as equal targets, the loss function assigns higher weights to pixels with smaller certified radii. This forces the optimizer (Projected Gradient Descent) to prioritize energy on the pixels most likely to change labels, maximizing the impact of the perturbation budget.
2.3 Black-Box Attack: CR-PBGD
For scenarios where model gradients are unavailable, we frame the attack as a Bandit Optimization problem. We propose the CR-PBGD (Certified Radius-Guided Bandit Gradient Descent) attack, which incorporates two key components:
- Two-Point Gradient Estimator (TPGE): Unlike one-point estimators that are often noisy or zeroth-order methods that are query-intensive, TPGE uses two queries per round to provide a provably unbiased and stable gradient estimate.
- Guided Search: The certified radius weights are integrated into the bandit framework, allowing the black-box attack to focus its limited query budget on the most susceptible regions of the image.
3. Experimental Evaluation
3.1 Setup
The framework was evaluated on three major segmentation benchmarks:
- Pascal VOC: General object segmentation (21 classes).
- Cityscapes: Urban road-scene understanding (19 classes).
- ADE20K: Complex scene understanding (150 classes).
Attacks were tested against state-of-the-art architectures, including PSPNet, PSANet, and HRNet.
3.2 Key Results
- Superior Effectiveness: In white-box settings, CR-PGD achieved up to a 50% relative gain over standard PGD in reducing pixel accuracy. Across all tested models and datasets, the average relative gain was 13.9%.
- Black-Box Efficiency: CR-PBGD demonstrated a much faster decrease in pixel accuracy compared to standard bandit attacks as the number of queries increased. The attack achieves a sublinear regret bound of $O(\sqrt{T})$, ensuring asymptotic optimality.
- Computational Overhead: The process of calculating the certified radius is efficient, adding only approximately 11% to the total computation time compared to standard PGD. This is achieved by using a limited number of samples and updating the vulnerability map at specific intervals.
- Resilience to Defenses: The framework remains effective even against models protected by specialized defenses like FastADT (adversarial training) and SEGCERTIFY. For instance, on the Pascal VOC dataset, a model defended by FastADT still experienced an accuracy drop to 41% under the CR-guided attack.
4. Discussion and Implications
This work highlights a critical irony in adversarial machine learning: the metrics designed to certify safety can be inverted to facilitate more potent attacks. The stability of the TPGE estimator in black-box settings suggests that for complex tasks like segmentation, the reliability of the gradient estimate is often more important than the raw speed of queries.
However, some limitations remain. The current framework is primarily optimized for untargeted attacks, aiming to maximize general misclassification rather than forcing a specific incorrect label. Additionally, while the attack is query-efficient, the thousands of queries required for bandit optimization might still be detectable by sophisticated API monitoring systems.
5. Conclusion
By leveraging the certified radius to identify pixel-wise vulnerabilities, this framework provides a new standard for evaluating the robustness of image segmentation models. The results demonstrate that surgical, guided perturbations are far more effective than uniform attack strategies, posing a significant challenge for current defensive measures in safety-critical computer vision.
Summary updated