Surface-Bound Attacks on Feed-Forward Distractor Separation

A new family of 3D reconstruction models rebuilds a scene from a handful of photos in a single forward pass, with no per-scene tuning — milliseconds instead of the minutes or hours that per-scene methods like Neural Radiance Field optimization demand. When the photos contain passers-by, moving cars, or shifting shadows, the newest of these models go one step further: in that same pass, they decide which parts of the scene are permanent geometry and which are transient intruders, and rebuild only the permanent part. Distractor-free Generalizable 3D Gaussian Splatting (DGGS) makes this decision almost entirely by checking cross-view consistency — a feature is judged static if it projects to the same place in every camera. That makes the static/transient decision a security boundary, and it is currently undefended: nobody has asked whether it can be moved by an adversary.

We propose to attack it with physically realizable perturbations. The 3D adversarial literature gives us the two halves of the weapon and no one has joined them. Generating Multi-view Adversarial Examples for Visual Geometry Grounded Transformer (MVAP-G) shows that a feed-forward generator can produce multi-view-coordinated perturbations that make a foundation reconstruction model collapse its point cloud to the origin — but its perturbations live in pixel space, on strictly static scenes, and are not physically realizable. Cage-Based Deformation for Transferable and Undefendable Point Cloud Attack (CageAttack) shows that low-frequency deformations of an object's actual surface are stealthy to humans, survive geometric defenses that destroy high-frequency noise, and transfer across architectures — but it attacks only static CAD-object classifiers via slow per-scene optimization. The intersection — surface-bound, multi-view-consistent perturbations aimed at the static/transient decision of a feed-forward reconstruction model operating in a dynamic scene — is empty in the literature, and it is where the most safety-relevant failure lives.

Core Claim

The claim is that cross-view geometric consistency, used as evidence of permanence, can be forged or destroyed by smooth deformations of real scene surfaces, causing two opposite catastrophes: an intruder can be baked into the permanent map (a ghost pedestrian etched into a robot's world model), or genuine static geometry can be pruned as a distractor (a hole where a load-bearing wall used to be). Formally, given a feed-forward reconstruction model $f$, input images $I$ of a dynamic scene, and a sparse cage of vertices $V$ bound to a target surface through barycentric coordinates (each surface point expressed as a weighted combination of nearby cage vertices, as in CageAttack), we seek cage displacements $\Delta V$ rendered into all views by a differentiable renderer $g$:

$$ \max_{\Delta V} \; \ell\big(f(g(I, V + \Delta V))\big) \quad \text{s.t.} \quad \|\Delta V\|_\infty \le \epsilon, \;\; \mathrm{TV}(\kappa) \le \tau $$

where $\ell$ is one of two attack losses (bake or prune), $\epsilon$ bounds per-vertex displacement, and $\mathrm{TV}(\kappa)$ is the total variation of curvature of the deformed surface, a smoothness bound borrowed in spirit from the curvature regularizers of Spacetime Surface Regularization for Neural Dynamic Scene Reconstruction (4DRegSDF). The curvature bound is what makes the perturbation physically plausible — a warp, not a corruption. A second, defensive claim is tested in the same program: adding surface-geometry evidence (per-primitive normals and planarity) to the static/transient decision restores robustness, which would resolve a live tension in the literature between coarse and fine distractor reasoning.

Research Program

The first deliverable is a feed-forward attack generator. Following MVAP-G's architectural lesson — a generator with cross-view attention, trained across many scenes so it needs no per-scene optimization (50–500 ms per scene versus 25–200 seconds for iterative attacks) — we build a network that consumes multi-view images of a dynamic scene and outputs cage-vertex offsets for a designated target surface, rendered into every view through differentiable Gaussian splatting so the perturbation is consistent by construction. The critical difference from MVAP-G is the space of the attack: MVAP-G's Dense Prediction Transformer head emits per-pixel noise that exists only in the image; our head emits a 3D warp that a physical actor could realize — stretch a banner, reshape a sign, apply a printed conformal skin. And unlike pixel noise, which alters appearance statistics and can trip the segmentation and uncertainty machinery that NeRF On-the-go and DGGS lean on, a geometric warp preserves appearance while lying about geometry, attacking exactly the signal the decision uses.

The second deliverable is the threat-model evaluation. We attack DGGS white-box, then test transfer of the same perturbations to other feed-forward pipelines built on visual geometry transformers, including snapshot-style dynamic predictors in the spirit of BTimer: Bullet-Time Feed-Forward Splatting, and measure both attack modes: how often transient-object Gaussians are retained as static, and how often static geometry is pruned. Transfer matters more than white-box success: CageAttack's low-frequency deformations transferred to a different classifier at 70% where point jitter fell below 20%, and we hypothesize the same physics — smooth warps change the data, not the weights — will carry over here. Pixel attacks, by contrast, typically overfit the white-box model.

The third deliverable is a defense, and with it a genuine open question. The distractor-removal literature disagrees with itself on granularity: 3DGS-HPC: Distractor-free 3D Gaussian Splatting with Hybrid Patch-wise Classification and RobustNeRF: Ignoring Distractors with Robust Losses find that coarse 16×16 patches are the sweet spot and that fine per-pixel reasoning is fragile, while 4DRegSDF and DynaSurfGS: Dynamic Surface Reconstruction with Planar-based Gaussian Splatting show exact surface sampling and normal/planarity regularizers stabilize dynamic geometry — but only through per-scene optimization with Jacobians and Hessians. We will test whether a feed-forward surface prior — a lightweight head predicting per-Gaussian normals and planarity scores, in the spirit of DynaSurfGS's planar Gaussians — added to DGGS's distractor scoring hardens the decision against warp attacks without reintroducing the fragility that coarse patches were chosen to avoid. Either outcome is publishable: if surface cues help, coarse-patch reasoning was the vulnerability; if they hurt, that is a negative result the field needs to hear.

Baselines

The do-nothing reference is feed-forward 3D Gaussian Splatting with no distractor handling at all: MvSplat applied directly to distractor-contaminated scenes, which the DGGS paper reports at 15.45 dB PSNR on the Statue scene of the RobustNeRF benchmark. It establishes how badly the static assumption fails unaided. The strongest existing method on this exact question is DGGS itself, at 21.02 dB PSNR on the same Statue scene — the only published feed-forward system that separates distractors without per-scene optimization — and it is the primary target; every attack will be run at matched perturbation budgets against it under the identical protocol, and every defense will be trained on the same data with the same compute and tuning budget. As a per-scene quality ceiling we cite RobustNeRF at 30.87 dB on its own benchmark, noting it belongs to a different latency class (minutes of per-scene optimization, cf. 13–17 minutes for 3DGS-HPC) and cannot serve an online perception stream; it calibrates the fidelity gap, not the deployment question. On the attack side, the strongest competitor is MVAP-G, whose multi-view-aligned pixel perturbations achieve a Chamfer distortion of 2.047 on Co3D where a universal 2D perturbation achieves only 0.659; we will run it against DGGS at matched rendered-image perturbation budgets. Two ablations isolate our key ingredient: the same generator architecture with the cage head replaced by an unconstrained pixel-noise head (isolating the surface constraint), and random smooth cage deformations of matched magnitude (isolating optimization from geometry). Defense baselines are statistical outlier removal and implicit-surface filtering: Extending Adversarial Attacks and Defenses to Deep 3D Point Cloud Classifiers shows outlier removal annihilates unconstrained noise (JSMA success 40.9% to 0.5%) at only a 2.6% clean-accuracy cost, while CageAttack retains 25–30% success even against the stronger IF-Defense. Beating the baselines means, concretely: a decision-flip rate (intruder retained, or static geometry pruned) at matched human-detectability that exceeds the pixel-space head and the random-warp ablation, with the defended DGGS recovering its clean-scene PSNR to within 0.5 dB of 21.02 dB under attacks that cost the unhardened model 3 dB or more.

Why not just…?

Why not just run MVAP-G pixel perturbations against DGGS? This is the most serious alternative and we treat it as a co-equal baseline rather than arguing around it. Pixel attacks are proven fast and devastating on static reconstruction, and their cross-view alignment might well forge the consistency signal DGGS trusts. Three considerations temper but do not settle it: pixel noise is per-model and historically transfers poorly, while low-frequency geometric deformations transferred at 70% versus under 20% for point jitter in CageAttack; pixel noise is not physically realizable, so it does not model the threat an autonomous vehicle actually faces from a warped surface in the world; and pixel noise perturbs appearance, which the segmentation-refinement and uncertainty machinery in DGGS and NeRF On-the-go can in principle flag, whereas a warp perturbs only geometry. The head-to-head experiment at matched budgets and matched human detectability is the decisive check; if pixel attacks flip the static/transient decision as effectively as warps, we will report that the surface constraint contributes only physical realizability and the proposal's defensive arm becomes the main result.

Why not just use per-scene robust optimization (RobustNeRF, RobustSplat, DeSplat)? Because the question is not fidelity but latency and threat surface. Per-scene methods need 10–45 minutes per scene — 3DGS-HPC alone takes 13–17 minutes on an RTX 4090 — while the models we attack run in 150 ms (BTimer) to a few hundred milliseconds. An online robot or vehicle perception stream cannot pause to optimize, and a per-scene method that reconstructs a clean static world still had to make a static/transient decision that is itself attackable. We include RobustNeRF's 30.87 dB as a fidelity reference, not a competitor on the deployment question.

Why not just post-filter the predicted Gaussians with statistical outlier removal? The literature already rules this out for exactly our perturbation class. Outlier removal destroys isolated high-frequency noise — that is where its 40.9%-to-0.5% result comes from — but CageAttack's smooth structural deformations survive even the stronger IF-Defense at 25–30%, and RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS shows transient objects spawn dense, locally coherent Gaussian clouds that pass density thresholds precisely because they are not outliers. Worse, Liu et al. document that such filters destroy thin, sparse, legitimate geometry. A baked-in intruder produced by our attack is the antithesis of a statistical outlier.

Why not just mask moving objects with an off-the-shelf 2D segmenter before reconstruction? 3DGS-HPC documents semantic mismatch — segmenters find "person" but miss moving shadows, shifted furniture, and debris — and Overcoming the Static Scene Assumption shows a single mislabeled frame propagates into permanent 3D artifacts. It is also beside the point: our attack perturbs geometry that segmentation correctly segments, lying about where the surface is rather than what it is. We will still run a SAM-style masking plus MvSplat baseline on clean scenes so the fidelity comparison is honest, but it cannot address the adversarial question by construction.

Why not just adversarially train DGGS and skip the attack study? Adversarial training needs an attack generator to train against, and the per-scene iterative attacks it would otherwise rely on take 25–200 seconds per instance — infeasible for curating training data at scale. Liu et al. showed adversarial training helps point-cloud classifiers but at real clean-accuracy cost. Our feed-forward generator is precisely the missing tool: it makes attack data cheap enough to train on, and the defense arm of the proposal closes that loop explicitly.

Proposed Experiments

We evaluate on three test beds from the file, chosen so that ground truth for the static scene exists. The RobustNeRF benchmark (four indoor distractor scenes plus Kubric, including the Statue scene with a ground-truth static background) is the primary setting: it lets us measure, in novel-view PSNR/SSIM/LPIPS, whether the intruder was baked in or static geometry was deleted, relative to clean DGGS at 21.02 dB on Statue. The DGGS synthetic distractor dataset (Re10K/ACID with COCO-injected distractors, with controllable distractor size and frequency) gives controlled sweep conditions; Co3D provides the bridge to the MVAP-G comparison, where the reported numbers to beat are Chamfer distortions of 2.047 (multi-view-aligned) and 0.659 (universal 2D) on VGGT. Beyond PSNR we report a decision-flip rate (fraction of transient Gaussians retained and static Gaussians pruned, measured against the ground-truth static scene), Chamfer distance of the reconstructed static geometry against ground truth, per-scene attack runtime, and a human plausibility study modeled on CageAttack's, where 64% of participants rated its deformations most plausible — our warps should be at least as hard to spot.

Four checks decide the proposal. First, transfer: perturbations optimized white-box on DGGS must degrade held-out feed-forward models; if transfer matches pixel-space transfer (the 10–30% black-box range typical of unconstrained attacks), the physics argument fails. Second, stealth: attacked outputs must pass statistical outlier removal and surface-reconstruction filters, with the CageAttack 25–30% IF-Defense survival rate as the precedent to clear. Third, the defense arm must restore clean-scene fidelity under attack to within 0.5 dB of DGGS's 21.02 dB, and must not degrade it on clean scenes. Fourth, the granularity tension: we compare hardened models with per-primitive surface scores against the coarse 16×16 patch scoring that 3DGS-HPC and RobustNeRF advocate, under both clean and attacked conditions — the first direct test of whether surface precision is a robustness asset or a fragility in feed-forward distractor separation. If pixel attacks win the head-to-head, we will say so plainly and reframe the contribution around the defense and the tension result, which would stand independently.