Don't Look into the Sun: Adversarial Solarization Attacks on Image Classifiers

Paul Gavrikov, Janis KeuperarXiv 2023 · DOI · arXiv

Abstract

Deep Neural Networks (DNNs) are notoriously brittle when faced with "out-of-distribution" (OOD) data. This paper introduces Adversarial Solarization, a novel attack that bridges the gap between imperceptible noise-based perturbations and fixed common corruptions. By exploiting the "shape-texture gap," this method preserves global object structure—keeping images recognizable to humans—while significantly degrading the performance of state-of-the-art image classifiers.

Introduction

Current methods for testing model robustness typically fall into two categories: mathematically rigorous but often unrealistic $\ell_p$ noise-based adversarial attacks, and realistic but non-adaptive common corruptions (e.g., ImageNet-C). Adversarial Solarization offers a middle ground: it is a computationally efficient, adaptive attack that specifically targets a model's unique weak points while maintaining the global semantic structure of the input.

Methodology

The core of the attack is the Solarization Function. For any pixel in an image, if its intensity exceeds a threshold $\alpha$, the value is inverted ($1 - \text{pixel}$); otherwise, it remains unchanged.

Visualizing Adversarial Solarization: Original images (top) vs. attacked images (bottom). Note how global structure is preserved while local textures are inverted.

Because the resulting loss landscape is highly non-convex and noisy, standard gradient-based optimization methods like PGD often fail. To address this, the authors proposed RandSol, a greedy random search algorithm. RandSol iteratively samples $\alpha$ values to find the threshold that most effectively flips the model's prediction. The study evaluates this attack on the ImageNet dataset across a wide range of architectures, from legacy models like VGG-16 to modern backbones like ConvNeXt and Vision Transformers (ViT).

Key Insights: The Shape-Texture Gap

Solarization highlights a fundamental divergence between human and machine vision. Humans primarily recognize objects by their silhouettes and global structure, which solarization preserves. In contrast, DNNs rely heavily on local color and texture distributions, which solarization completely disrupts.

Even models trained with advanced data augmentation like RandAug—which includes solarization—are not immune. The attack can still identify specific "weak spot" thresholds that the training process failed to cover. Furthermore, the authors found that robustness against standard corruptions does not necessarily translate to resilience against adversarial solarization, suggesting that current benchmarks may be missing critical dimensions of vulnerability.

Results

The impact of the RandSol attack is devastating. A standard ResNet-50 model saw its Top-1 accuracy plummet from 76.13% to just 13.48%. While modern architectures using robust training recipes (such as those from the timm library) showed more resilience, they still suffered significant drops; for instance, a timm ResNet-50 dropped to 64.77%.

Test accuracy across varying solarization thresholds for legacy (left) and modern (right) models.

Interestingly, the attack is most effective at flipping the primary prediction (Top-1), while the correct label often remains within the model's Top-5 guesses. This suggests that the model retains some global understanding of the object but is ultimately distracted by the solarized local features.

Conclusion

Adversarial Solarization exposes a significant vulnerability in how modern classifiers process visual information. While solarization-aware training provides some defense, no model achieved full immunity. Future research should investigate how these attacks affect other tasks like object detection and semantic segmentation, where local pixel values are critical for defining object boundaries.

Summary updated

Abstract

Assessing the robustness of deep neural networks against out-of-distribution inputs is crucial, especially in safety-critical domains like autonomous driving, but also in safety systems where malicious actors can digitally alter inputs to circumvent safety guards. However, designing effective out-of-distribution tests that encompass all possible scenarios while preserving accurate label information is a challenging task. Existing methodologies often entail a compromise between variety and constraint levels for attacks and sometimes even both. In a first step towards a more holistic robustness evaluation of image classification models, we introduce an attack method based on image solarization that is conceptually straightforward yet avoids jeopardizing the global structure of natural images independent of the intensity. Through comprehensive evaluations of multiple ImageNet models, we demonstrate the attack's capacity to degrade accuracy significantly, provided it is not integrated into the training augmentations. Interestingly, even then, no full immunity to accuracy deterioration is achieved. In other settings, the attack can often be simplified into a black-box attack with model-independent parameters. Defenses against other corruptions do not consistently extend to be effective against our specific attack. Project website: https://github.com/paulgavrikov/adversarial_solarization
BibTeX
@misc{gavrikov2023don,
  author = {Paul Gavrikov and Janis Keuper},
  title = {{Don't Look into the Sun: Adversarial Solarization Attacks on Image Classifiers}},
  year = {2023},
  eprint = {2308.12661},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.12661},
}
Figure