All You Need is RAW: Defending Against Adversarial Attacks with Camera Image Pipelines

Yuxuan Zhang, Bo Dong, Felix Heide

Abstract

Existing neural networks for computer vision tasks are vulnerable to adversarial attacks: adding imperceptible perturbations to input images can fool these methods into making false predictions. While various defense methods propose image-to-image mapping or adversarial training, they often ignore the fact that natural RGB images are recovered from RAW color filter array captures. In this work, we exploit the RAW data distribution as an empirical prior for adversarial defense. We propose a model-agnostic defensive method that maps input RGB images to Bayer RAW space and back to output RGB using a learned camera image signal processing (ISP) pipeline to eliminate adversarial patterns. Our method acts as an off-the-shelf preprocessing module that generalizes to unseen tasks without additional retraining. Experiments on ImageNet, COCO, and Pascal VOC validate that our method significantly outperforms existing input-transformation defenses.

Introduction

Deep Neural Networks (DNNs) are notoriously fragile when faced with adversarial attacks—small, carefully crafted perturbations that are invisible to humans but catastrophic for model accuracy. Most existing defenses fall into two categories: Adversarial Training (AT), which is computationally expensive and often overfits to specific attacks, and Input Transformation (IT), which attempts to "denoise" images before they reach the model.

However, most IT methods operate entirely in the RGB space, ignoring the fundamental truth that natural images are recovered from RAW sensor data through a complex ISP pipeline. Adversarial noise typically does not follow the statistical properties of real sensor data, such as photon shot noise or optical aberrations. This makes the RAW domain a powerful space for identifying and eliminating artificial patterns.

Conceptual diagram of the defense mechanism: mapping adversarial RGB images ($Y'$) back to the RAW domain ($X$) and reconstructing clean RGB images ($Y$).

Methodology

We propose a "Raw-Defense" preprocessing module that acts as a bridge between the adversarial RGB input and the clean RAW-like distribution. The pipeline consists of three main components:

  1. F Operator (Image-to-RAW Mapping): A learned encoder-decoder network that maps an input RGB image ($I'$) back to its estimated intermediate RAW measurements ($I_W$). It is trained to map both clean and noisy images to the same RAW distribution.
  2. G Operator (Learned ISP): A pyramidal convolutional neural network (based on PyNet) that reconstructs a high-quality RGB image ($I_G$) from the estimated RAW data. It uses a coarse-to-fine structure to handle different scales of image features.
  3. S Operator (Conventional ISP): A traditional, non-differentiable software ISP (performing demosaicing, white balance, etc.) to produce a second RGB reconstruction ($I_S$).

The final "defended" image is a weighted sum of the outputs from $G$ and $S$. The inclusion of the non-differentiable S operator is a critical security measure; it breaks the gradient flow, making it significantly harder for an attacker to generate "white-box" adversarial examples that can bypass the defense.

The complete Raw-Defense pipeline, including the F, G, and S operators, visual examples of intermediate images, and the detailed architecture of the G operator.

Experimental Results

Our method was validated on standard benchmarks including ImageNet (classification), MS COCO (semantic segmentation), and Pascal VOC (object detection). The $F$ and $G$ operators were trained on the Zurich-Raw-to-RGB dataset.

Key Findings

  • Superior Robustness: Against strong PGD attacks on ImageNet, our method maintained ~66% Top-1 accuracy on ResNet-101, while competitors like JPEG compression and Total Variation Minimization (TVM) often dropped below 50%.
  • The "Rawer" the Better: Defense performance is highest when the intermediate mapping space is closest to the actual sensor output. Mapping to a "demosaiced" stage instead of "raw capture" leads to a drop in defensive efficacy.
  • Real Statistics Matter: The defense fails if trained on synthetic RAW data (e.g., simple downsampling). This proves the method exploits unique statistical cues found in real physical sensors rather than just performing simple blurring.
  • Task Generalizability: As a model-agnostic module, Raw-Defense requires no knowledge of the downstream model. A pipeline trained on street scenes successfully defended models for ImageNet classification and Pascal VOC detection without retraining.

Conclusion

By leveraging the natural distribution of RAW sensor data as an empirical prior, Raw-Defense effectively "washes away" adversarial perturbations. This approach provides a robust, task-agnostic, and off-the-shelf defense mechanism that outperforms traditional RGB-based input transformations. Future work will explore using RAW natural image statistics as an unsupervised prior to potentially remove the need for paired training data.

Summary updated

Abstract

Existing neural networks for computer vision tasks are vulnerable to adversarial attacks: adding imperceptible perturbations to the input images can fool these methods to make a false prediction on an image that was correctly predicted without the perturbation. Various defense methods have proposed image-to-image mapping methods, either including these perturbations in the training process or removing them in a preprocessing denoising step. In doing so, existing methods often ignore that the natural RGB images in today's datasets are not captured but, in fact, recovered from RAW color filter array captures that are subject to various degradations in the capture. In this work, we exploit this RAW data distribution as an empirical prior for adversarial defense. Specifically, we proposed a model-agnostic adversarial defensive method, which maps the input RGB images to Bayer RAW space and back to output RGB using a learned camera image signal processing (ISP) pipeline to eliminate potential adversarial patterns. The proposed method acts as an off-the-shelf preprocessing module and, unlike model-specific adversarial training methods, does not require adversarial images to train. As a result, the method generalizes to unseen tasks without additional retraining. Experiments on large-scale datasets (e.g., ImageNet, COCO) for different vision tasks (e.g., classification, semantic segmentation, object detection) validate that the method significantly outperforms existing methods across task domains.
BibTeX
@inproceedings{zhang2022all,
  author = {Yuxuan Zhang and Bo Dong and Felix Heide},
  title = {{All You Need is RAW: Defending Against Adversarial Attacks with Camera Image Pipelines}},
  booktitle = {European Conference on Computer Vision},
  year = {2022},
  url = {https://openreview.net/forum?id=rKqYNLf2US},
}
Figure