Red-Teaming Robot Policies on Raw Scans Without Mesh Repair

Robots are increasingly tested by searching for object shapes that make them fail. Geometric Red-Teaming for Robotic Manipulation is the clearest example. It deforms 3D object meshes into "CrashShapes" that drop a learned insertion policy's success from 90% to 22.5% in simulation and on real robots. Its deformation model, however, only works on watertight, cleanly connected meshes. The objects a robot actually meets arrive as raw depth scans: triangle soups with holes, overlapping pieces, and edges shared by three or more faces. The usual fix is to repair the scan into a clean surface first. That repair smooths away the millimeter-scale contact geometry that GRT showed matters most. It can also merge or split thin parts, which changes the object being tested.

We propose to run adversarial shape search directly on raw scans. The deformation is a smooth spatial warp that carries every raw triangle along, whatever its connectivity. Local rigidity is imposed only where the scan has material, meaning inside the object according to a winding-number test that tolerates holes and overlaps. A connectivity guard rejects "attacks" that win by thinning a handle or peg until it would snap or could not be fabricated. The claim is testable in two parts. Attacks found this way should be as effective as attacks on repaired meshes. More of them should also survive the checks a physical adversarial object must pass: it stays in one piece, keeps its handles as loops, meets a minimum wall thickness, and still breaks the policy when 3D-printed.

Core Claim

Two properties make an adversarial deformation physically meaningful. The object must be locally rigid, so it looks like a plausibly manufactured variant of the original. It must also be topologically intact, so it has the same number of connected pieces and handles, with no part thinned toward breaking. The existing tools for both properties assume clean connectivity that raw scans lack.

Classical as-rigid-as-possible (ARAP) energy penalizes how far each small neighborhood departs from a pure rotation. It needs edge weights assembled from mesh connectivity. ARAPReg states that its learned regularizer requires a single fixed manifold topology. DEMEA needs a clean template and precomputed skinning weights. Topology-aware losses have a matching limitation: clDice, as used by Lv et al., and persistence-barcode matching in TopoMatch are defined only on regular pixel or voxel grids.

Our position is that neither property has to be computed on the scan's own connectivity. Both can be computed on a volume that the raw scan defines. We define that volume with the generalized winding number $w_M(x)$ of the raw triangle soup $M$ at point $x$. This is a soft inside/outside score that remains meaningful for open and overlapping surfaces, and it is the tool that Fast and Globally Consistent Normal Orientation based on the Winding Number Normal Consistency uses to orient normals on unoriented input. The adversary optimizes a smooth warp $\phi_\theta$ of 3D space, and the deformed asset is $\phi_\theta(M)$. The objective is

$$ \min_\theta \; \mathcal{L}_{\text{task}}\big(\phi_\theta(M)\big) \;+\; \lambda_r \int_{\Omega(M)} \min_{R \in SO(3)} \big\| \nabla \phi_\theta(x) - R \big\|_F^2 \, dx \;+\; \lambda_t \, \mathcal{L}_{\text{conn}}\big(\phi_\theta, \Omega(M)\big), $$

where the terms are as follows. $\mathcal{L}_{\text{task}}$ rewards failure of the victim policy or perception model. $\Omega(M) = \{x : w_M(x) > 1/2\}$ is the scan's material region. $\nabla\phi_\theta$ is the warp's Jacobian, and the middle term is a volumetric ARAP energy that is zero exactly when the warp is locally a rotation inside the object. $\mathcal{L}_{\text{conn}}$ is a soft-skeleton connectivity loss on the warped occupancy, defined below. The weights $\lambda_r$ and $\lambda_t$ trade attack strength against realism.

Restricting the rigidity integral to $\Omega(M)$ is what separates this from applying a generic elastic penalty to a deformation field. It lets empty gaps, such as the air between a mug body and its handle, deform freely, while the material itself stays rigid.

Research Program

The deformation space. We compare two parameterizations of $\phi_\theta$. The first is a cage. Following CageNet, which wraps wild meshes in a manifold envelope and moves signals between envelope and mesh with mean value coordinates (a smooth interpolation scheme), we build the cage and move its vertices. CageNet used the cage only for feedforward segmentation and skinning. We use it as the set of variables an attacker optimizes.

The second is a small coordinate network whose Jacobian is regularized. This follows the elastic penalty that Nerfies applies to deformation fields in dynamic radiance fields, but masked to $\Omega(M)$. The cage gives a low-dimensional space suited to gradient-free search. The network gives finer spatial control. Which one works better is an empirical question, and we treat it as an ablation.

Handle and anchor points. GRT asks a vision-language model which points to perturb and which to fix. We keep that interface so that the only differences between our method and GRT are the representation and the regularizer.

The connectivity guard. We rasterize the warped occupancy $w_M(\phi_\theta^{-1}(x))$ onto a coarse grid, around $64^3$ over the object's bounding box. On that grid we apply the soft centerline intersection from clDice between the warped and original skeletons. This penalizes an optimizer that severs a thin branch or pinches a neck.

Full differentiable persistent homology is not used in the loop. TopoMatch-style barcode matching has cubic cost in grid size, and TopoMatch is formulated for 2D images. We therefore use persistence only as an exact after-the-fact check. On a fine grid at print resolution we count connected components and handles (Betti numbers $b_0$ and $b_1$) and measure minimum wall thickness. We cannot use ray-based thickness here, because on cracked scans the rays escape, as Fast and robust shape diameter function documents.

Rendering and simulation. Differentiable rasterizers draw each triangle independently, so raw soups can be rendered directly. The connectivity barrier on raw data lies in the regularizers and in collision handling, not in rasterization. For contact simulation we give the engine the winding-number signed distance of the warped scan, which is well defined without repair.

Victims and optimizers. Where the victim is a vision model, gradients flow through rendering. Where the victim is a contact-rich policy, we use GRT's population-based optimizer (TOPDM) over the same variables, so the comparison with GRT stays like-for-like.

Baselines

Undeformed raw scan. The naive reference is the raw scan itself, placed into the task. This measures how much of any failure comes from scan artifacts rather than adversarial deformation. It matters because raw scans may already degrade policies trained on clean CAD models.

Repair-then-GRT. The strongest existing method is GRT run on a repaired version of the same scan. The repair is a watertight wrap or voxel remesh, followed by GRT's Jacobian-field deformation and TOPDM search. The final CrashShape is evaluated both as the repaired mesh and after its deformation is transferred back to the raw scan.

The reference numbers come from GRT on clean YCB and IndustReal assets. Insertion success fell from 90% to as low as 22.5%. Blue-team fine-tuning recovered up to 60 percentage points on CrashShapes while retaining 87–96% on nominal objects. For reference on the nominal task, IndustReal reports 82–99% success in simulation. No reported result was found on Lacuna for GRT, or any geometric red-teaming method, on raw non-manifold scans.

For fairness, both methods use the same handle and anchor points, the same number of simulator rollouts, the same population size, and the same tuning budget for regularization weights. Beating this baseline means one of two outcomes. Our method matches the policy-success drop within the confidence interval while producing a significantly higher fraction of attacks that pass the topology, thickness, and fidelity checks. Or it achieves a larger drop at equal validity.

Laplacian-smoothed deformation. TT3D, Towards Transferable Targeted 3D Adversarial Attack in the Physical World, optimizes grid-based radiance-field appearance and makes small Laplacian-smoothed mesh adjustments. It reaches about 89% targeted black-box attack success, against about 5% for its mesh baseline, and up to 91% on printed prototypes. Its authors found that strong geometric smoothing limited attack effectiveness. We run the same warp with a Laplacian penalty on the repaired mesh in place of masked ARAP, under the same budget. This isolates whether rigidity, rather than smoothness, is the ingredient that keeps attacks both strong and realistic.

Ablations. Three ablations test the key ingredients. The first removes the connectivity guard. The second applies the rigidity integral over the whole bounding box instead of $\Omega(M)$. The third swaps the cage for the coordinate network.

Why not just…?

Why not just repair the scan and run GRT as published? This is the main threat, and the repair-then-GRT baseline tests it directly. The evidence that it may fall short is indirect. CageNet reports that aggressive remeshing strips detail and invalidates fine annotations. Subdivision-based Mesh Convolution Networks argues that downstream results are capped by repair quality. GRT itself found that small curvature changes at contact points cause most failures. None of these shows that repair hides adversarial vulnerabilities specifically.

The experiment will show whether it does. If repaired CrashShapes, transferred back to raw scans and printed, fail the robot as often as ours, the raw-scan machinery is unnecessary for this task, and we will report that plainly.

Why not just put an elastic penalty on a deformation field everywhere, with no mask and no topology term? This is the Nerfies recipe, and it needs no connectivity at all, so it is a genuine competitor. Its predictable weakness is that rigidity enforced in empty space couples parts that are separated only by air. That coupling suppresses legitimate deformations, such as opening a handle's gap. The same weakness produces the "mitten" failure that CageNet reports for coarse cages. The unmasked ablation measures whether this matters. If the unmasked field matches masked ARAP on every metric, we drop the winding-number mask.

Why not just use Laplacian smoothing or total variation, as TT3D and the 3D cloak work do? ARAPReg reports that Laplacian-type penalties shrink shapes and collapse thin parts during articulation. With rigidity instead, it cut reconstruction error by 16.8% on DFAUST, 16.6% on SMAL, and 6.7% on Bone. TT3D found that its Laplacian regularizer pushed the attack toward texture instead of geometry. Both findings point the same way, but neither was measured in an adversarial loop on scans. The Laplacian baseline supplies that measurement.

Why not skip the topology guard and filter invalid shapes afterwards? Filtering discards optimizer effort, and it can leave only degenerate winners if the easiest attack is always to thin a peg. The no-guard ablation reports two things: the fraction of successful attacks that fail the Betti-number or thickness checks, and the effective attack rate after filtering. If filtering alone gives the same valid-attack rate, the guard is not needed.

Proposed Experiments

Assets. We build raw meshes from the RGB-D scans in the YCB Object and Model Set, using the 22 grasping objects from GRT. We also scan the IndustReal peg and hole parts. Both methods are run on the identical raw input.

Tasks. The first task is IndustReal insertion with its published policies in Isaac Gym, followed by real-robot evaluation of printed attacks on a Franka Panda, mirroring GRT's protocol. The second task tests the gradient-based path. We attack multi-view classifiers through differentiable rendering, following TT3D's IM3D evaluation. Here the reference is TT3D's roughly 89% black-box targeted success on clean reconstructions.

Metrics. We measure the drop in policy success, or attack success rate for classifiers. We measure distortion as the average deviation of the warp Jacobian from a rotation inside the material. Fidelity is the Chamfer and normal error of the deformed raw scan against the undeformed scan outside the handle region. Topological validity is whether $b_0$ and $b_1$ are preserved at print resolution and whether minimum wall thickness stays above the printer's limit. Finally, we measure the real-robot failure rate of printed shapes.

Decisive checks. Three checks decide the claim. First, at matched simulator budget, our method must produce more attacks that are both valid and effective than repair-then-GRT. Second, printed versions of those attacks must fail the physical robot at a rate comparable to their simulated rate. Third, the ablations must show that each of the mask, rigidity instead of smoothness, and the connectivity guard changes at least one of these outcomes.

Blue-teaming transfer. A final experiment asks whether policies fine-tuned on repaired-mesh CrashShapes stay robust to raw-scan CrashShapes, and the reverse. A gap in either direction would show that mesh repair shifts which failures a red-team finds.