Adversarial Cast Shadows: Stealth Attacks That Live in Light Transport, Not on Surfaces

Every physical adversarial attack in the current literature modifies the target object itself: a camouflage wrap on a car, a sticker on a stop sign, a painted patch on the asphalt. That is a liability. Modified surfaces must survive human inspection, printer gamut limits, and specularity shifts, and the most successful defenses look exactly for anomalous surface patterns. This proposal develops a different attack surface: the scene's lighting. We use hybrid lighting representations — which split illumination into a smooth ambient field plus sharp directional components, and into distant sky versus nearby indirect light — as the machinery for optimizing cast shadows: adversarial perturbations that live entirely in shadows thrown by physically mundane objects, leaving every surface in the scene pristine and unpainted.

The enabling observation comes from two literatures that have not met. Hybrid lighting work can now render differentiable, spatially varying illumination and crisp shadow boundaries: the street-scene model of Neural Light Field Estimation for Street Scenes with Differentiable Virtual Object Insertion combines a high-dynamic-range sky dome with a volumetric grid of Spherical Gaussian lobes — compact directional "spikes" of light — and renders differentiable ground-plane shadows; NeFII: Inverse Rendering for Reflectance Decomposition with Near-Field Indirect Illumination adds a near-field cache for local light bounces. Meanwhile, adversarial stealth work has established that the most durable attacks are the ones that look like ordinary physics: Beware of Road Markings: A New Adversarial Patch Attack to Monocular Depth Estimation hides perturbations as painted road markings, and Adversarial Wear and Tear: Exploiting Natural Damage for Generating Physical-World Adversarial Examples hides them as scratches and dirt. A shadow is a stronger member of that family than any of them: shadows are unregulated, expected at every sun angle, and no surface carries forensic evidence of the attack.

Core Claim

We claim that an adversary can plant an ordinary-looking roadside structure — a perforated shade panel, a lattice fence, a planter with slats — whose cast shadow pattern on the road and on passing vehicles is optimized, through a differentiable renderer built on a hybrid lighting representation, to degrade a vehicle detector, while (i) the structure itself passes inspection as benign street furniture, (ii) no target surface is touched, and (iii) the attack remains effective across the range of sun positions the scene will actually experience, because the optimization samples sun angles from a physically plausible lighting manifold rather than an unconstrained parameter space.

The closest prior work attacks incident light rather than transport. Adversarial Relighting Against Face Recognition optimizes a single point light (physically realized with a robotic arm) against face recognition under a Lambertian reflectance model — no cast shadows, no spatial variation, and a narrow portrait domain. Adversarial Geometry and Lighting using a Differentiable Renderer optimizes low-order Spherical Harmonics lighting coefficients against image classifiers, but because spherical harmonics act as a low-pass filter on the sphere, the authors were forced to assume matte surfaces and to drop cast shadows entirely. Ours differs on the axis that matters: the perturbation is not a change in the light source but a change in what the light hits on its way to the sensor, modeled by exactly the frequency- and distance-decomposed machinery (ambient field plus directional lobes; distant dome plus near-field bounces) that the prior attacks could not represent. Among physical camouflage attacks, RAUCA: A Novel Physical Adversarial Attack on Vehicle Detectors via Robust and Accurate Camouflage Generation and 3D Gaussian Splatting Driven Multi-View Robust Physical Adversarial Camouflage Generation treat lighting purely as a nuisance to average over, and both optimize albedo on the target's surface; we treat lighting as the attack medium and leave the target's surface untouched.

Research Program

Three components get built. First, a differentiable occlusion renderer for outdoor street scenes, assembled from existing hybrid pieces: the sky-dome-plus-volumetric-Spherical-Gaussian lighting model and differentiable ground-plane ratio shadow map of Neural Light Field Estimation for Street Scenes with Differentiable Virtual Object Insertion, extended with a near-field bounce term in the spirit of NeFII so that shadowed ground and vehicle panels receive the correct local inter-reflections rather than pure ambient darkness. Second, an occluder optimizer: the free parameters are the geometry of a benign structure (slat positions, perforation pattern, curvature of a canopy), and the objective is detection loss of a monocular detector under rendering of the shadowed scene. Third, a lighting-manifold constraint: sun positions, intensities, and ambient levels are sampled from plausible configurations — anchored to the parametric lighting distributions modeled by Deep Parametric Indoor Lighting Estimation, Editable Indoor Lighting Estimation, and EverLight: Indoor-Outdoor Editable HDR Lighting Estimation — so the optimized shadow works for sun angles the scene will see, not one baked configuration. This addresses the known failure mode that physical attacks degrade under environmental shift, which Relighting Scenes With Object Insertions in Neural Radiance Fields and Rendering-Aware HDR Environment Map Prediction from a Single Image identify as the cost of single-representation lighting: soft components alone cannot render crisp shadow edges, and sharp components alone cannot fill the penumbra naturally.

The optimization, in words: sample a plausible sun position and ambient state, render the scene with the candidate occluder through the hybrid lighting model, feed the image to the detector, and backpropagate the detection loss through the renderer into the occluder's geometry, with a term penalizing deviation from the plausible lighting manifold. Formally, with occluder parameters $\theta$, scene geometry $\mathcal{G}$, sun configuration $s$ drawn from a plausible set $\mathcal{S}$, differentiable renderer $R$, and detector $D$:

$$\max_{\theta}\;\mathbb{E}_{s\sim\mathcal{S}}\;\ell\big(D\big(R(\mathcal{G},\theta,s)\big)\big)$$

where $\ell$ is the detection loss (e.g., the detector's localization confidence on the target vehicle class).

Proposed Experiments

The primary benchmark is the CARLA multi-weather setting used by RAUCA and PGA, attacking the same white-box YOLOv3 detector, evaluated over the same distances (5–20 m) and pitch angles (20°–60°) that PGA reports, plus a held-out sweep of sun angles that neither prior attack evaluates. The headline metric is [email protected] on the target class; secondary metrics are attack persistence as a function of time-of-day (sun azimuth/elevation), and a naturalness evaluation pitting shadow-attacked frames against surface-camouflage frames in a two-alternative forced-choice human study, following the stealth tradition of Adversarial Camouflage: Hiding Physical-World Attacks With Natural Styles. A physical phase follows the established protocol: a 1:12 scale model of the road segment with the fabricated occluder, photographed under a movable point light simulating sun traversal, mirroring RAUCA's 552-photograph physical evaluation. On the defensive side, we retrain the detector on shadow-attacked synthetic insertions using the augmentation pipeline of Neural Light Field Estimation for Street Scenes, which showed a 2% mean-average-precision gain from hybrid-lit virtual object insertion on nuScenes, to measure how much adversarial-shadow augmentation closes the hole.

Baselines

The do-nothing reference point is the clean scene with an unmodified benign occluder present: RAUCA reports a clean-vehicle [email protected] of 0.70 for YOLOv3 on the CARLA multi-weather white-box setting, which is the number any attack must drag down. The strongest existing competitor is RAUCA's surface camouflage, which drops that same detector to [email protected] of 0.304 in simulation (versus ~0.44 for the ACTIVE competitor), and to AP 0.137 in physical night-time tests where competitors never fell below 0.24; we re-run RAUCA's released camouflage in our CARLA splits with the same detector, the same weather set, and an optimization budget matched in GPU-hours and iterations to ours, so any difference is attributable to the attack medium rather than tuning. PGA is the second surface-camouflage competitor; the file states it "consistently outperformed state-of-the-art physical attack methods in digital experiments" but no exact CARLA number is recoverable, so we say: no reported result found on Lacuna for PGA's numerical AP values, and we will re-run it under the same matched budget. A third baseline is the road-marking-style patch attack of Beware of Road Markings, the closest precedent for ground-plane perturbations that mimic street furniture. The key ablation isolates the hybrid representation itself: the same shadow-occluder optimization run with (a) spherical-harmonics-only ambient lighting and no directional lobe, and (b) the full hybrid model with near-field bounces. Beating RAUCA means dropping [email protected] below 0.304 in simulation while leaving every target surface unmodified, and sustaining the drop across the sun-angle sweep where surface camouflage demonstrably weakens; the ablation counts as evidence for the hybrid claim only if (a) is markedly worse than (b).

Why not just…?

Why not just attack the light source with low-order spherical harmonics, as Liu et al. already did? Because the spherical-harmonics parameterization is exactly what rules out shadows: it is a low-pass filter on the sphere, and Adversarial Geometry and Lighting using a Differentiable Renderer explicitly had to assume matte surfaces and local illumination with no cast shadows to keep gradients tractable. Our ablation (a) runs precisely this comparison, and the hybrid literature predicts soft blurred penumbrae from the harmonics-only variant — both weaker as an attack and less natural to a human who knows what a hard-edged noon shadow looks like. We will report this honestly even if the gap is small.

Why not just use a plain 2D HDR environment map, which already contains sharp and smooth light in one image? A single environment map has no spatial extent: it describes light arriving from infinity, so it cannot model how illumination changes across a street or how a nearby structure re-shades a passing vehicle. Empirically, the street-scene paper measured median sun-position angular errors of 24.88° to 53.86° for environment-map-style estimators versus 22.43° for the hybrid sky-dome-plus-volumetric-Gaussian model — and sun position error converts directly into misplaced shadows. For our attack, a wrong sun means the optimized shadow lands in the wrong place at deployment time.

Why not just train over random lighting variation with Expectation over Transformations, like RAUCA does across 16 weathers? This is the most serious alternative, and we treat it as such: EoT is cheap and it demonstrably helps. But it treats light as a nuisance to become invariant to, and the residual is visible in RAUCA's own numbers — its physical night-time AP of 0.137 versus its simulation 0.304 shows that robustness acquired by averaging does not transfer cleanly to unseen lighting regimes. More fundamentally, EoT cannot produce a shadow attack at all: it has no mechanism for placing a perturbation in the shadow, because the perturbation it optimizes is on the target's surface. Our experiment settles the composition question directly — we test a surface camouflage trained with EoT plus our adversarial shadow, expecting the combination to beat either alone, and expecting our shadow-only attack to hold up on the sun-angle sweep where EoT-trained surface camouflage decays.

Why not just use a black-box neural renderer like RAUCA's Neural Renderer Plus to handle lighting implicitly? Because the attack must be realized physically, and a neural renderer's parameters do not correspond to any physical quantity. RAUCA's renderer blends environmental features into an image, but there is no invertible path from its internal features back to a sun elevation, a shadow-casting structure, or a fabrication plan — the file's own analysis notes that such approximators cannot enforce physical constraints like printability or surface reflectance. Our occluder parameters are geometry, which a laser cutter can execute exactly.

Decisive Checks

The proposal lives or dies on three results. The simulation check: shadow-only perturbations drop white-box YOLOv3 [email protected] below RAUCA's 0.304 on matched CARLA settings. The robustness check: the drop persists across a sun-angle sweep spanning a plausible day, while surface-camouflage baselines degrade. The stealth check: human raters judge shadow-attacked frames no more suspicious than clean frames, and rate them less suspicious than surface camouflage. If the sun-angle sweep instead shows the attack collapsing outside its training range, the lighting-manifold sampler has failed and the core claim is wrong — that would itself be the paper's finding, since no prior work has measured how lighting geometry, not weather, governs physical attack persistence.