DMT-Attack: Diffusion-Prior Texture Search for Closed-Loop Vision-Language-Action Policies

Jun Xia, Shangwen Wang, Jun Ma, Ruichao Ren, Zhouyang Jia, Xin Zhao, Huijun Liu, Jie YuarXiv 2026 · arXiv

TL;DR

DMT-Attack is a novel adversarial attack that uses a generative diffusion model to search for natural-looking textures (like tablecloth patterns) that disrupt robot manipulation tasks. By optimizing in the latent space of a diffusion model rather than raw pixels, it achieves a 57.9% attack success rate while remaining significantly stealthier than traditional "digital noise" patches.

What Problem Does This Paper Address?

Vision-Language-Action (VLA) policies are the "brains" of modern robots, allowing them to follow instructions like "pick up the bowl" by processing continuous visual feedback. However, these policies are vulnerable to adversarial attacks—specifically, persistent textures placed in the environment that confuse the robot's perception.

The core challenge is that most existing adversarial attacks produce highly conspicuous, non-natural patterns. While effective at causing a robot to fail, these "digital noise" patches (see Figure 1B) are easy for a human observer to spot and remove. The authors of this paper aim to create "grounded" attacks: textures that look like ordinary scene objects (like a patterned tablecloth) but are mathematically optimized to break the robot's decision-making loop.

Figure 1. DMT-ATTACK targets the VLA perception–action loop using persistent textures. (A) The policy uses visual observations, language, and robot state to perform a manipulation task. (B) A conventional patch introduces a conspicuous local pattern. (C) A diffusion prior generates a tabletop texture whose content is searched for task-disruptive effects.

Methods

The authors propose DMT-Attack, a gradient-free "gray-box" attack that leverages a frozen Stable Diffusion 2.1 model as its texture generator. Instead of tweaking pixels, the attack searches the diffusion model's latent space for task-disruptive candidates.

The architecture (detailed in Figure 2) consists of two main phases:

  1. Phase I: Teacher Initialization: The system is initialized using "teacher" latents—previously discovered successful attack patterns—to jumpstart the search process.
  2. Phase II: Closed-Loop Refinement: An eight-expert latent proposal network suggests new textures. These are passed through an appearance filter to ensure they look natural (checking for things like color entropy and frequency noise) before being rendered in a simulator.
  3. Feedback Loop: The robot's performance is evaluated in a "closed-loop" rollout. The attack uses Proximal Policy Optimization (PPO) to update the proposal network based on rewards derived from:
  4. Attention Displacement: How much the attack distracts the robot's visual attention.
  5. Action/Trajectory Deviation: How much the robot's movements differ from a "clean" run.
  6. Task Failure: Whether the robot ultimately fails the instruction.
Figure 2. Main architecture and learning procedure. Phase I distills high-gap teacher latents into an eight-expert text-conditioned proposal family using matching, diversity, and load-balancing objectives. In Phase II, a frozen diffusion model decodes the selected latent into a candidate texture; appearance-screened candidates are evaluated by a frozen VLA in closed-loop rollouts, and paired behavioral signals update the proposal, critic, and exploration policy. The diffusion model, renderer, and VLA remain frozen.

Datasets

The researchers evaluated their method using the LIBERO benchmark, a standard for lifelong robot manipulation. They tested across four distinct suites:

  • LIBERO-Object: Manipulating various small objects.
  • LIBERO-Spatial: Tasks emphasizing spatial relationships.
  • LIBERO-Goal: Reaching specific goal states.
  • LIBERO-Long: Multi-step, long-horizon tasks.

They tested these against two prominent VLA policies: $\pi_{0.5}$ and OpenVLA-OFT.

Key Insights and Ideas

  • Generative Models as Search Spaces: The most powerful idea here is using a diffusion prior to constrain the attack. By searching within the distribution of "natural images" learned by Stable Diffusion, the attack is forced to produce textures that look like real-world patterns.
  • Closed-Loop is Crucial: Unlike static image classifiers, robots act over time. An effective attack must disrupt the entire temporal sequence of actions. DMT-Attack uses rollout feedback to ensure the texture remains disruptive throughout the whole task.
  • Multi-Expert Proposals: Using eight independent "expert" heads allows the system to explore different "families" of disruptive textures simultaneously, making the search more robust and efficient.
  • The Stealth-Effectiveness Trade-off: There is a fundamental tension between how much an attack breaks a model and how visible it is. DMT-Attack pushes the Pareto frontier, achieving high disruption with low visibility (see Figure 3).
Figure 3. Method-level attack and appearance summary. The ver- tical coordinate reports macro cASR from the independent-state main experiment; the horizontal coordinate reports 1−MMD from the separate CLIP texture-collection evaluation.

Key Results

  • Superior Disruption: DMT-Attack achieved a macro clean-conditioned attack success rate (cASR) of 57.9%, significantly higher than the best baseline (UPA-RFAS at 44.7%) and far above random natural textures (19.7%).
  • High Stealth: In anomaly detection tests, DMT-Attack was flagged only 45% of the time, whereas traditional optimized patches were flagged 97.5% to 100% of the time.
  • Visual Fidelity: The textures achieved much higher CLIP similarity scores to clean scenes compared to baselines, proving they "blend in" better to the environment.
  • Generalization: Textures found on one set of tasks often transferred effectively to others, suggesting they exploit fundamental vulnerabilities in how VLAs process visual scenes.

Limitations and Future Work

  • Simulation-Only: The current work is performed entirely in the MuJoCo simulator. While the textures look natural, the "sim-to-real" gap remains a hurdle for deploying such attacks (or defenses) in the physical world.
  • Gray-Box Requirements: The attack requires access to some internal signals (like attention maps) to calculate rewards during the search phase, though it doesn't need gradients.
  • Defense Implications: The authors suggest that future robot defenses shouldn't just look for "weird pixels" but must jointly monitor visual appearance and behavioral anomalies to catch these more sophisticated, grounded attacks.

Summary updated

Abstract

Vision-language-action (VLA) policies rely on visual feedback throughout manipulation, making them vulnerable to persistent texture attacks. However, optimized patterns can stand out from their surroundings, making an attack easier to notice and its carrier more likely to be inspected or removed. To address this limitation, our key insight is to search the structured image distribution of a diffusion model for natural-looking textures while using closed-loop feedback to identify task-disruptive candidates. Based on this idea, we propose DMT-Attack, a gradient-free gray-box attack that generates textures for ordinary scene objects such as tablecloths. A frozen text-conditioned diffusion model defines the search space, and an eight-expert latent proposal network is initialized from evaluated teacher latents and refined using rollout feedback. We evaluate DMT-Attack against pi_0.5 and OpenVLA-OFT across four LIBERO suites. DMT-Attack achieves a macro clean-conditioned attack success rate (cASR) of 57.9%, compared with 44.7% for UPA-RFAS and 19.7% for random natural textures. It also attains higher scene similarity and a lower clean-calibrated detector flag rate (45% versus 97.5-100% for optimized patch baselines). These results demonstrate that natural-looking textures can disrupt VLA visual feedback and closed-loop decision-making, motivating defenses that jointly assess visual appearance and task behavior.
BibTeX
@misc{xia2026dmt,
  author = {Jun Xia and Shangwen Wang and Jun Ma and Ruichao Ren and Zhouyang Jia and Xin Zhao and Huijun Liu and Jie Yu},
  title = {{DMT-Attack: Diffusion-Prior Texture Search for Closed-Loop Vision-Language-Action Policies}},
  year = {2026},
  eprint = {2611.25695},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25695},
}
Figure