Rigidity Under Invisible Light: Stress-Testing ARAP Tracking Against Cross-Modal Optical Attacks

Cameras do not see what we see. Human vision is blind to ultraviolet and infrared, cannot follow light flickering faster than about 60 Hz, and cannot perceive the polarization of light — yet silicon sensors register all three. A growing line of physical adversarial attacks exploits exactly this: invisible near-infrared laser speckle on traffic signs, UV-emitting hats that blind face recognition, and LED flicker that a rolling-shutter camera converts into stripes that fool image classifiers, all while a human bystander sees nothing unusual. Separately, a mature line of work in 3D vision relies on as-rigid-as-possible (ARAP) regularization — a prior stating that although a surface may bend globally, every small patch of it should move like a rigid plate. This prior is what lets systems track cloth, skin, and animal bodies as they deform, instead of melting or tearing the reconstructed mesh. The two lines have never met: every ARAP-regularized tracker assumes a benign sensor, and every cross-modal attack targets 2D classifiers or static single-frame 3D regression.

This proposal makes the first systematic study of what happens at that meeting point, and argues the answer is not obvious in either direction. An ARAP prior is a low-pass filter on deformation, so it might quietly absorb rolling-shutter stripes or polarization-induced "ghost" normals as kinematically implausible. But it is also an edge-coupled energy: a corrupted observation at one vertex pulls on its neighbors, so a local sensor perturbation could instead be spread into a globally coherent, physically plausible false deformation — the most dangerous kind, because it passes the very check meant to certify plausibility. We will characterize both regimes, build an attack that is optimized through the ARAP term rather than around it, and design a robustified rigidity prior that cuts the propagation channel.

Core Claim

The claim is twofold. First, ARAP regularization in dynamic 3D tracking is not a defense against cross-modal optical attacks: an attacker who optimizes the injected light pattern against the full tracking objective — photometric data term plus ARAP energy — can find perturbations that the rigidity prior actively accepts, because a rolling-shutter stripe pattern maps naturally into per-image-band rigid translations, which are exactly what ARAP permits. Second, a small modification to the prior fixes this: replacing the squared edge residuals in the ARAP energy with a robust penalty and gating the data term per surface element turns the prior from a propagation channel into a genuine filter. The claim is falsifiable: if the characterization study shows the standard prior already suppresses all attack waveforms, the first half is wrong and we will report that as the finding.

The closest existing work makes the boundary precise. Fooling Polarization-Based Vision Using Locally Controllable Polarizing Projection shows that a modified LCD projector can project adversarial polarization angles that trick shape-from-polarization networks into hallucinating 3D "ghost geometries" on flat surfaces — but only against static, single-frame normal regression, with no temporal tracking and no rigidity prior. 4DTAM: Non-Rigid Tracking and Mapping via Dynamic Surface Gaussians is the strongest template-free deformable tracker and applies an explicit ARAP loss to its neural warp field — but it assumes a completely benign RGB-D sensor. Invisible Perturbations: Physical Adversarial Examples Exploiting the Rolling Shutter Effect provides the differentiable analytic model of stripe formation we reuse, but it attacked a 2D ResNet-101 image classifier, never a 3D tracker. Nobody has asked the interaction question, and the answer determines whether the growing deployment of ARAP priors in robotics and surgical navigation (Nerfies, 4DTAM) inherits any robustness from its physical assumptions.

Research Program

The program has three parts built on a common testbed: the 4DTAM tracker, run on the synthetic Sim4D benchmark (50 animated meshes with ground-truth sensor trajectories, depth maps, and surface normals) plus real rolling-shutter captures of deforming cloth, with perturbations injected using the analytic row-exposure stripe model of Sayles et al. and, secondarily, polarization-corrupted normals in the style of Li et al. fed through a Deep Shape from Polarization-style front end.

The first part is a characterization study. We sweep LED frequency (just above the 60 Hz human flicker-fusion threshold up to 1 kHz), phase offset, and amplitude, and independently sweep the ARAP weight $\lambda$ in the tracker, mapping where the prior suppresses the stripes and where it converts them into warp-field oscillation, mesh distortion, or trajectory drift. Note that frequency close to 60 Hz combined with a roughly 33 ms frame readout yields only a few wide bands — a low spatial frequency perturbation, which matters for the filtering alternatives discussed below. This map is the paper's empirical backbone and answers the open question that both 4DTAM and Sayles et al. leave untouched.

The second part is the new attack. Because the stripe model is differentiable end-to-end into the tracker's objective, we optimize the LED waveform to maximize tracking error subject to the physical invisibility constraint (frequency above 60 Hz, exposure faster than the roughly 1/750 s stripe-formation threshold reported by Sayles et al.). Our hypothesis, grounded in the structure of ARAP, is that the optimizer converges on waveforms whose induced deformation decomposes into per-band rigid motions — deformation the prior certifies as physical. We call this rigid-band ghost deformation, and it is the key novel ingredient: prior cross-modal attacks (Invisible Reflections, Omni-Angle Assault, The Fluorescent Veil) all optimize against a fixed 2D network; none optimizes against a geometric regularizer.

The third part is the defense. The standard objective has the form $E = L_{\text{data}} + \lambda L_{\text{ARAP}}$ with squared residuals in both terms. We robustify it:

$$E = \sum_i g(r_i)\, d_i \; + \; \lambda \sum_{(j,k)\in\mathcal{E}} \rho\!\left(\big\| \tilde{R}_{jk}(\mathbf{v}_j-\mathbf{v}_k) - (\mathbf{v}'_j-\mathbf{v}'_k) \big\|^2\right),$$

where $d_i$ is the photometric residual of surface element $i$, $r_i$ its temporal consistency residual, $g$ a down-weighting gate, and $\rho$ a Huber-type penalty on the failure of edge $(j,k)$ to map its rest-shape segment $\mathbf{v}_j-\mathbf{v}_k$ to its deformed segment under the locally fitted rigid rotation $\tilde{R}_{jk}$. The intent is that corrupted elements are quarantined by $g$ and corrupted edges by $\rho$, so a local perturbation cannot borrow rigidity from its neighbors. This directly extends the elastic-regularization tradition of Nerfies (which penalizes the deviation of the deformation Jacobian's singular values from unity with a quadratic penalty) and the structural rigidity of DEMEA (which restricts deformation to rotations and translations on a coarse embedded graph), while drawing on the evidence from ARAPReg and NeuroMorph that unconstrained or merely Laplacian-smoothed alternatives melt and tear.

Baselines

The do-nothing reference point is unmodified 4DTAM under attack with no defense of any kind, which quantifies the raw degradation the rest of the study explains and mitigates. On clean data this system reports an Absolute Trajectory Error of 0.25 cm on the Sim4D "curtain" sequence, against 6.10 cm for the rigid-prior baseline SurfelWarp (4DTAM); those two numbers anchor both the clean ceiling and the level of failure that counts as a successful attack.

The strongest existing competitor is 4DTAM itself in its published configuration, since it is the only system that combines template-free non-rigid tracking with an explicit ARAP loss and is evaluated on Sim4D; any claim that our modifications matter must beat it under attack while matching it on clean data. Fairness means identical input streams, identical optimization iterations and compute (the published system runs at roughly 1.5 frames per second, so compute is matched by wall-clock iteration budget), and an identical tuning budget — we grid-search $\lambda$ and all defense hyperparameters on clean held-out Sim4D sequences for every variant, spending the same search effort on each. Beating it means holding clean ATE within a small tolerance of 0.25 cm on curtain-type sequences while reducing attacked ATE by a large factor; a defense that wins only by degrading clean performance does not count.

Two defense baselines bracket the alternatives a reviewer would name. One is spatial low-pass filtering of the input frames before tracking, tuned on the same validation sequences. The other is a detection gate in the spirit of the Spatial Frequency Analysis detector of Sato et al., which reports 96% detection accuracy against infrared laser speckle on static traffic signs: we retrain an equivalent frequency-domain detector on our deforming scenes and reject or down-weight flagged frames. Finally, the key ablation isolates the proposal's central ingredient by running 4DTAM with the ARAP loss removed ($\lambda = 0$) and with the ARAP loss replaced by Laplacian smoothing: if the attack succeeds mainly when the ARAP term is present, the propagation mechanism is confirmed; if it succeeds equally without it, the story reduces to ordinary photometric corruption and the claim must be revised.

Why not just…?

Why not just low-pass filter the frames? This is the most serious alternative, and the literature already warns against it twice: Sato et al. and Sayles et al. both note that passive spatial filtering destroys legitimate high-frequency content — sign glyphs, cloth weave, facial creases — degrading clean performance. More pointedly, an LED frequency just above the 60 Hz flicker-fusion threshold produces only a few wide bands per frame, i.e., a low-spatial-frequency perturbation that a low-pass filter passes untouched. Our attack optimizer is free to find such waveforms, and experiment three runs the low-pass baseline head-to-head; if a fixed kernel fully restores ATE at no clean cost, that result stands and we would report the problem as solved by trivial means.

Why not just detect the attack, as Sato et al. do at 96% accuracy? The 96% figure was obtained on static, planar traffic signs where any high-frequency anomaly is suspicious. A deforming scene is intrinsically full of temporal and spatial frequency content — that is what the tracker exists to model — so the detector's false-positive behavior on legitimate motion is unknown, and our optimizer explicitly seeks waveforms inside the motion band the detector cannot flag without flagging the scene itself. We treat this honestly: the detector baseline is run, and if it gates the attack at acceptable false-positive rates on dynamic scenes, it wins and we say so.

Why not just switch to global-shutter cameras? For the rolling-shutter modality specifically, this is a complete fix, and Sayles et al. concede as much. But the deployment reality is the opposite: casual-capture systems like Nerfies are built explicitly around commodity phone cameras, and 4DTAM runs on consumer RGB-D hardware, for cost and sensor-density reasons. More importantly, global shutter does nothing to the polarization and non-visible-spectrum modalities, so the general question — does a geometric prior defend a dynamic tracker against out-of-band optical injection? — survives any single hardware swap. The defense we propose targets that general question.

Experiments and Decisive Checks

Experiment one produces the failure-mode map on Sim4D: attacked ATE and surface alignment error as a function of LED frequency, amplitude, and $\lambda$, with clean 4DTAM's 0.25 cm curtain ATE as reference. The decisive readout is the sign of the interaction: if attacked error decreases as $\lambda$ grows, the prior is a filter and the field inherits robustness for free; if it increases or plateaus at high error, the prior propagates. Experiment two runs the through-the-regularizer attack: success is defined as attacked ATE exceeding the SurfelWarp failure level of 6.10 cm (a more-than-twentyfold degradation from 0.25 cm) while the waveform stays above 60 Hz and evades the frequency detector at its operating point — for calibration, Sayles et al. achieved an 84% targeted physical attack success rate on a 2D classifier under comparable constraints, and Li et al. dropped glass-segmentation IoU from 0.715 to 0.361 with a 100% invisible projection, so a successful 3D-tracking attack would be the first of its kind. Experiment three evaluates the robustified energy: it counts as a win if clean ATE stays within roughly 1.5× of 0.25 cm while attacked ATE under the optimized waveform falls below one-tenth of the undefended value, beating both the low-pass and detector-gate baselines under the same tuning budget, with the $\lambda=0$ and Laplacian-only ablations showing how much of the recovery is specifically attributable to the robust rigidity term. Experiment four is a physical demonstration: an LED array flickering above 60 Hz illuminating deforming cloth captured by a rolling-shutter RGB camera, with marker-based ground truth, testing whether the simulated rigid-band ghost deformation survives real optics, ambient light dilution (the known weakness of every optical attack in this literature, from the 6000-lux washout of Omni-Angle Assault to the sunlight dilution of Invisible Reflections), and readout noise. A secondary stretch goal repeats the pipeline with polarization-corrupted normals on the 25-object Deep Shape from Polarization capture set, whose clean mean angular error of 18.52° versus 41.44° for the preceding physics-only method (Deep Shape from Polarization) gives a measured handle on how much normal corruption enters the tracker; no reported result found on Lacuna for dynamic SfP tracking, which is exactly the hole this fills.

The decisive check for the whole proposal is simply whether the interaction exists at all. Either answer is publishable and useful: a rigidity prior that already filters cross-modal optical attacks would tell the robotics and surgical-navigation communities that their ARAP terms are worth more than they thought, and one that propagates them — our expectation — would identify a new, humanly invisible attack surface on exactly the systems that manipulate cloth and tissue, together with a fix that costs one change of penalty function.