RefCloth: Transferable Adversarial Tablecloths for Vision-Language-Action Models

Shaozhen Liu, Zhiwei Li, Jiayang Sun, Shuangjun Du, Yi Zhong, Shenrui Ma, Yaozu Liu, Qi Li, Zhenan SunarXiv 2026 · arXiv

TL;DR

RefCloth is a novel adversarial attack framework that generates a single, persistent "adversarial tablecloth" texture capable of disrupting Vision-Language-Action (VLA) models. Unlike previous attacks that require hijacking camera feeds or modifying target objects, RefCloth works zero-shot across unseen tasks and different robot policies simply by being present in the workspace.

What Problem Does This Paper Address?

Vision-Language-Action (VLA) models are the "brains" of modern generalist robots, mapping visual observations and text instructions directly to motor commands. These models typically rely on multiple camera views—usually a static "third-person" view of the scene and a "wrist" view attached to the robot's arm.

While we know these models are vulnerable to visual perturbations, existing attacks have significant practical hurdles:

  1. High Privilege: Some attacks (like 2D patches) require "hijacking" the digital sensor stream to overlay noise, which is hard to do in the real world.
  2. Object Specificity: Other attacks modify the texture of the objects the robot is supposed to grab. If the task changes or a new object is introduced, the attack must be redesigned and the model often retrained.

The authors of RefCloth ask: Can we create a passive, low-privilege attack that lives in the environment (like a tablecloth) and transfers across different tasks and robot models without any retraining?

Methods

The core of RefCloth is optimizing a single 3D texture for a tablecloth that remains effective even as the robot moves and occludes parts of it.

The RefCloth optimization pipeline uses a matched-reference branch to cancel out rendering artifacts.

Key technical innovations include:

  • Persistent Scene Carrier: Instead of a floating 2D patch, the attack is bound to a 3D surface in the world. This ensures it appears consistently (though from different angles) in both the main and wrist cameras.
  • Surface-Augmented Optimization: To prevent the texture from overfitting to a single robot trajectory, the authors apply random rotations, reflections, and translations in the texture's UV coordinate space during training. This "UV diversification" makes the pattern disruptive regardless of how it's sliced by the camera's view.
  • Matched-Reference Calibration: This is perhaps the most clever bit. Differentiable renderers (used to get gradients for optimization) often have slight "static offsets" compared to the high-fidelity native simulators used for testing. RefCloth routes a "clean" version of the cloth through the exact same rendering path to create a baseline. By maximizing the displacement from this matched reference rather than a generic one, they cancel out rendering artifacts that would otherwise hinder zero-shot transfer.

Datasets

The researchers evaluated RefCloth using the LIBERO benchmark, a standard for lifelong robot learning. They tested across four distinct suites:

  • LIBERO-Spatial: 10 tasks focused on spatial relationships.
  • LIBERO-Object: 10 tasks involving different manipulation objects.
  • LIBERO-Goal: 10 tasks with varying goal conditions.
  • LIBERO-Long: 10 multi-step long-horizon tasks.

They used OpenVLA-OFT as the source model for optimization and tested transferability to other models like $\pi_0$ and $\pi_{0.5}$.

Key Insights and Ideas

  • The Wrist Camera is a Major Weak Point: Because the wrist camera moves closer to the workspace during manipulation, the tablecloth often covers a much larger percentage of its field of view (up to ~55%) compared to the main view (~14%).
  • Synergistic Disruption: The attack is more than the sum of its parts. Disrupting the wrist view alone causes a significant drop in success, but disrupting both views simultaneously leads to a synergistic collapse in performance that exceeds the combined individual drops.
  • Learned Structure Matters: The attack isn't just high-contrast noise. When compared against "random" tablecloths with matched statistics (same brightness, contrast, and smoothness), RefCloth was significantly more effective, proving it learns specific spatial structures that confuse the VLA's internal representations.
  • Zero-Shot Transfer is Real: A texture optimized on a few tasks in the "Spatial" suite successfully disrupted the robot when it was moved to the "Object" suite—where the tablecloth was on the floor instead of a table—and even when the underlying robot policy was swapped for a different architecture.
RefCloth causes the robot to grasp the wrong objects or follow erratic paths.

Key Results

  • Massive Success Drop: On average across 40 tasks, the RefCloth texture reduced the success rate of the $\pi_0$ model by 62.50 percentage points and the $\pi_{0.5}$ model by 43.75 percentage points compared to a plain cloth.
  • Wrist vs. Main View: In ablation studies, attacking only the wrist view dropped success from 98% to 75%, while attacking both views crashed it to 34%.
  • Robustness to "Clean" Reference: Using the "Matched-reference" optimization improved the attack's effectiveness by about 12.5 percentage points on the $\pi_{0.5}$ policy compared to a standard "Native" reference.

Limitations and Future Work

While RefCloth is a powerful demonstration of VLA vulnerability, it has some limitations:

  • Simulation-Heavy: The work is primarily evaluated in simulation (MuJoCo and SAPIEN). While the "matched-reference" technique helps bridge the gap between different simulators, the "sim-to-real" jump to physical fabric remains a challenge.
  • Fixed Geometry: The attack assumes the tablecloth is a flat, non-deforming surface. Real-world cloth wrinkles and folds, which would change the UV mapping.
  • Future Directions: The authors suggest that extending this framework to handle real-world lighting, fabric deformation, and physical printing is the next step for auditing and hardening embodied AI policies.

In summary, RefCloth shows that we don't need to hack a robot's software to break it; sometimes, just changing the decor is enough.

Summary updated

Abstract

Closed-loop vision-language-action (VLA) policies rely continuously on multi-view cameras, yet existing visual attacks typically hijack sensor streams with 2D patches or alter target object meshes that require retraining when manipulation objects change. We introduce RefCloth, a framework that optimizes one persistent adversarial tablecloth on a frozen source policy and deploys it zero-shot across unseen tasks and target models without modifying task objects or cameras. Throughout manipulation, static third-person and moving wrist views observe this shared world-bound texture under evolving occlusions. To generalize across shifting surface views while bridging the gap between differentiable and native renderers, RefCloth combines bounded texture parameterization with shared surface UV transformations and a same-path clean reference that removes static rendering offsets at initialization. Across four LIBERO suites in native simulation, one frozen texture transfers from OpenVLA-OFT to $π_0$ and $π_{0.5}$, reducing success relative to an unoptimized cloth by 62.50 and 43.75 percentage points on average and far exceeding amplitude- and variation-matched random controls. Dual-view audits and target closed-loop camera interventions show that moving wrist exposure drives over twice the standalone drop of the main view with strong synergy, while ablations confirm the roles of UV diversification and path calibration.
BibTeX
@misc{liu2026refcloth,
  author = {Shaozhen Liu and Zhiwei Li and Jiayang Sun and Shuangjun Du and Yi Zhong and Shenrui Ma and Yaozu Liu and Qi Li and Zhenan Sun},
  title = {{RefCloth: Transferable Adversarial Tablecloths for Vision-Language-Action Models}},
  year = {2026},
  eprint = {2611.22736},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22736},
}
Figure