Bridging the Sim-to-Real Gap in Physical Adversarial Attacks

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2857 papers1,179 conceptsfrozen 2026-10-04

In the digital world, breaking a computer vision model is trivial. By adding a tiny, mathematically optimized "noise" to an image, an attacker can force a state-of-the-art neural network to see a toaster where there is a school bus. However, as Ivan Evtimov and colleagues demonstrated in Robust Physical-World Attacks on Machine Learning Models, these digital perturbations are incredibly brittle. When you print that noise and stick it on a real-world object, the attack almost always fails.

The reason is the sim-to-real gap. In a digital attack, the attacker has direct access to the pixels. In the physical world, those pixels must pass through a "physical renderer": they are affected by changing lighting, camera sensor noise, viewing angles, and atmospheric conditions. This research direction focuses on how to bridge this gap, creating adversarial objects that remain effective in the messy, unpredictable physical world.

The Physical World as a Differentiable Renderer

To create a physical attack that works, researchers have realized they cannot just optimize for a single static image. Instead, they must optimize for a distribution of possible views.

As surveyed by Haoran Wang and Alec Helbling in Adversarial Attacks Using Differentiable Rendering: A Survey, the key breakthrough is the use of differentiable rendering. By placing a 3D model of the target (like a car or a person) inside a digital simulation that mimics real-world physics, attackers can backpropagate through the rendering process itself. This allows them to optimize a texture or shape that is robust to variations in lighting, shadows, and perspective.

Modeling Non-Rigid Deformations

One of the hardest challenges in physical attacks is dealing with objects that change shape. A sticker on a stop sign is rigid, but a pattern on a T-shirt is constantly folding and wrinkling.

In Adversarial T-Shirt! Evading Person Detectors in a Physical World, Kaidi Xu and his team addressed this by using Thin Plate Spline (TPS) mapping. By mathematically modeling how fabric deforms as a person moves, they were able to optimize a "cloaking" pattern that remains effective even when the T-shirt is wrinkled. This approach tripled the effectiveness of the attack compared to methods that assumed a flat, static surface.

From Meshes to Neural Radiance Fields

Traditional 3D attacks often relied on manipulating the vertices of a mesh or the pixels of a texture. However, these can be difficult to translate to the physical world without losing precision. Recent work has moved toward more sophisticated 3D representations.

In Towards Transferable Targeted 3D Adversarial Attack in the Physical World, Yao Huang and Yinpeng Dong introduced TT3D. Instead of optimizing a mesh, they optimize the internal parameters of a Neural Radiance Field (NeRF). This allows the adversarial features to be "baked" into the 3D volume of an object, making the attack highly transferable across different viewpoints and even different types of vision models.

The Reality Gap and the Stealth Requirement

Despite these advances, a significant "reality gap" remains. Jakob Shack and Katarina Petrović found in Breaking the Illusion: Real-world Challenges for Adversarial Patches in Object Detection that even when digital lighting is perfectly matched to physical conditions, there can still be a 64% drop in effectiveness. This suggests that current simulators still fail to capture the complex interplay between light, material textures, and camera sensors.

Furthermore, as the field matures, researchers are looking beyond mere effectiveness. In Illusory Attacks: Information-theoretic detectability matters in adversarial attacks, Tim Franzmeyer argues that a successful physical attack must also be stealthy. If an adversarial pattern is so garish that it triggers a "safe mode" or alerts a human operator, the attack has failed its ultimate goal. This has led to new techniques like CNCA, which uses diffusion models to generate adversarial camouflage that looks like natural textures—such as zebra stripes or graffiti—to the human eye.

Closing the Loop with Photorealistic Simulation

The latest frontier in this research is the integration of high-fidelity game engines with deep learning frameworks. Mansi Phute and ShengYun Peng developed UNDREAM, a framework that bridges photorealistic simulators like Unreal Engine with differentiable renderers. By allowing gradients to flow from a complex, dynamic 3D world back to the adversarial texture, researchers can finally optimize for the true complexity of the physical environment, including changing weather and dynamic shadows.

Papers in this version

Showing 60 of 857 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 808 papers
  2. Version 1 · 2026-10-04 · 857 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_physical_world_adversarial_robustness_c42971e4,
  title = {physical-world adversarial robustness},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/physical-world-adversarial-robustness/rdr_c42971e47b334074942e9e1ab43535c5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_bridging_the_sim_to_real_gap_in_physical_c42971e4_v1,
  title = {Bridging the Sim-to-Real Gap in Physical Adversarial Attacks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bridging-the-sim-to-real-gap-in-physical-adversarial-attacks/rdr_c42971e47b334074942e9e1ab43535c5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}