Toward Robust and Accurate Adversarial Camouflage Generation against Vehicle Detectors

Jiawei Zhou, Linye Lyu, Daojing He, Yu Li

TL;DR

This paper introduces RAUCA (Robust and Accurate Camouflage Attack), a framework that generates full-body physical adversarial camouflage to deceive vehicle detectors across diverse viewpoints and weather conditions. By implementing a novel End-to-End Neural Renderer Plus (E2E-NRP) with a unique UV-traversal sampling method, the authors achieve true end-to-end texture optimization while realistically simulating environmental factors like lighting, rain, and fog.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) excel at vehicle detection for autonomous driving, but they are notoriously vulnerable to physical adversarial attacks—such as custom-printed vehicle wraps (camouflage) designed to make cars "invisible" to AI detectors.

To design effective camouflage, researchers use differentiable neural renderers to simulate how a 3D textured vehicle looks in a 2D image, allowing them to pass gradients back from the object detector to optimize the texture. However, current methods suffer from three major flaws:

  1. Mapping Inaccuracies: "World-aligned" methods project a 2D pattern onto a vehicle repeatedly, leading to discrepancies when the wrap is actually deployed. Conversely, existing "UV-map-based" methods fail to optimize the texture end-to-end, requiring non-differentiable post-processing steps that degrade attack performance.
  2. The "Noise" Problem: Standard renderers struggle to optimize every pixel on a vehicle's UV map due to sampling artifacts, resulting in discontinuous, noisy patterns.
  3. Weather Blindness: Most camouflage is optimized for clear, sunny days. When subjected to fog, rain, or night conditions, the adversarial effect quickly vanishes.
Comparison of different adversarial camouflages under sunny and foggy environments. Only RAUCA succeeds in both.

Methods

The authors introduce RAUCA, an advanced pipeline that bridges the gap between digital simulation and physical deployment.

Fig. 2. The overview of RAUCA. First, we create a multi-weather dataset using CARLA, which includes car images, corresponding mask images, and camera transformation sets. Then, the car images are segmented using the mask images to obtain the foreground car and background images. The foreground car image, the 3D model, and the camera transformation are passed through the E2E-NRP rendering component for rendering. The rendered image is then seamlessly integrated with the background. After a series of random output augmentation, the image is fed into the object detector. Finally, we optimize the adversarial camouflage through back-propagation with our devised loss function computed from the output of the object detector.

The core technical innovations include:

  • UV-Traversal Sampling: Traditional renderers use tensor-traversal sampling, projecting 3D facets back onto the 2D UV map. This leaves massive gaps of unoptimized pixels (appearing as random noise). RAUCA flips this logic: it traverses every point on the 2D UV map directly and maps it to the 3D facet tensor, ensuring 100% texture coverage and smooth, continuous adversarial patterns.
Fig. 3. The illustration of tensor-traversal sampling and its issue. The triangle represents a facet’s area in the UV map. (a) illustrates the tensor-traversal sampling method’s projection of a facet from Tfc onto the UV map. The projected positions of Tfc are represented by green points, while black points denote the pixel points on the UV map. (b) illustrates the issue of this sampling method: the red points are optimized during the camouflage generation, while the cyan points are not optimized. The numerous unoptimized points can lead to a decline in attack effectiveness. (c) is the desired situation in which all points on the UV map can be optimized. (d) shows the optimization result using tensor-traversal sampling, which leads to camouflage patterns resembling random noise with poor continuity. In contrast, (e) depicts the result of UV- travel sampling, exhibiting strong texture continuity and yielding more effective and robust attacks.
  • Environment Feature Extractor (EFE): To make the simulation realistic, an encoder-decoder network extracts environmental characteristics (shadows, lighting, weather) from background scenes and blends them pixel-by-pixel with the rendered vehicle foreground.
  • Accelerated & Pre-trained EFE: Training an EFE from scratch for every vehicle is computationally expensive. The authors accelerated this process by 13x using an offline-rendered image cache. Furthermore, they pre-trained the EFE on primitive shapes (cubes, cylinders) and a mix of vehicles, allowing the system to rapidly adapt to entirely unseen vehicle models (like planes or boats) with minimal fine-tuning.

Datasets

The authors leveraged the CARLA autonomous driving simulator to build two key datasets:

  • Base Multi-Weather Dataset: 40,960 images capturing 16 distinct weather conditions (combinations of 4 sun altitudes and 4 fog densities) across 128 viewpoints.
  • Enhanced Multi-Weather Dataset: 32,768 higher-quality images featuring random camera viewpoint perturbations and customized vehicle material properties (simulating the reflective properties of real-world vinyl stickers).
  • EFE Pre-training Dataset: 120,960 training and 58,368 testing images containing diverse objects (basic shapes, multiple car models, and a boat) to teach the renderer generalized environmental interaction.

Key Insights and Ideas

  • The Sampling Blindspot: The realization that standard neural renderers leave texture pixels unoptimized due to coordinate projection math was a major "aha" moment. Shifting from facet-traversal to UV-traversal directly translates to cleaner, more effective real-world camouflage wraps.
  • Cohesive Weather Simulation: Previous methods applied weather noise strictly to the vehicle foreground, creating a jarring foreground-background mismatch that broke the detector's gradient flow. Blending background environmental features into the foreground via the EFE ensures physical consistency.
  • Shape Agnostic Pre-training: Including basic geometric primitives (cubes, cylinders) in the EFE pre-training phase allows the network to learn how light wraps around arbitrary surfaces, making it highly transferable to completely novel vehicle classes.

Key Results

RAUCA-final was evaluated against six popular object detectors (including YOLOv3, YOLOX, and Faster R-CNN) in both simulation and physical world settings (using 1:12 scale models).

  • Simulation Performance: On the white-box YOLOv3 detector, RAUCA-final dropped the vehicle detection Average Precision ([email protected]) from 0.648 (Normal) down to 0.010 in seen weather, and to 0.012 in unseen weather (like heavy rain).
  • Black-Box Transferability: Against five black-box detectors, RAUCA-final outperformed the previous state-of-the-art method (ACTIVE) by an average margin of 14%, demonstrating excellent attack transferability.
  • Defense Resilience: RAUCA proved highly resilient against adversarial defense mechanisms like Local Gradients Smoothing (LGS) and Patch-Agnostic Defense (PAD) because it applies a full-body camouflage wrap rather than a localized patch.

Limitations and Future Work

Despite its strengths, RAUCA has a few notable limitations:

  • Conspicuousness: The generated textures feature highly saturated colors and abrupt pattern variations. While mathematically optimal for blinding AI, they are incredibly obvious and suspicious to human observers.
  • Static Assumptions: The framework optimizes camouflage using static images. It does not account for temporal dynamics encountered in the real world, such as motion blur or video-based target tracking systems.

Future Directions: The authors plan to integrate diffusion models to generate more natural-looking, stealthy textures that deceive both humans and machines. They also aim to extend the framework to video domains to explicitly attack temporal video trackers.

Summary updated

Abstract

Adversarial camouflage is a widely used physical attack against vehicle detectors for its superiority in multi-view attack performance. One promising approach involves using differentiable neural renderers to facilitate adversarial camouflage optimization through gradient back-propagation. However, existing methods often struggle to capture environmental characteristics during the rendering process or produce adversarial textures that can precisely map to the target vehicle. Moreover, these approaches neglect diverse weather conditions, reducing the efficacy of generated camouflage across varying weather scenarios. To tackle these challenges, we propose a robust and accurate camouflage generation method, namely RAUCA. The core of RAUCA is a novel neural rendering component, End-to-End Neural Renderer Plus (E2E-NRP), which can accurately optimize and project vehicle textures and render images with environmental characteristics such as lighting and weather. In addition, we integrate a multi-weather dataset for camouflage generation, leveraging the E2E-NRP to enhance the attack robustness. Experimental results on six popular object detectors show that RAUCA-final outperforms existing methods in both simulation and real-world settings.
BibTeX
@article{zhou2025toward,
  author = {Jiawei Zhou and Linye Lyu and Daojing He and Yu Li},
  title = {{Toward Robust and Accurate Adversarial Camouflage Generation against Vehicle Detectors}},
  journal = {IEEE Trans. Dependable Secur. Comput.},
  year = {2025},
  doi = {10.1109/tdsc.2025.3596311},
}
Figure