TL;DR
When you delete a sensitive object from a 3D scene, is it truly gone? This paper introduces Remove360, a real-world benchmark and four-part evaluation suite that reveals how state-of-the-art 3D Gaussian Splatting (3DGS) removal techniques often leave behind invisible "semantic residuals"—traces that humans miss, but modern vision foundation models can still detect and reconstruct.
What Problem Does This Paper Address?
As casual 3D capture apps become commonplace on smartphones and AR headsets, users increasingly need to erase private items—such as family photos, personal documents, or specific indoor decor—before publishing or sharing 3D reconstructions.
While 3D Gaussian Splatting (3DGS) makes deletion conceptually easy (simply prune the Gaussians corresponding to the target), current literature evaluates removal mostly through visual aesthetics or foreground segmentation. It ignores a critical privacy question: Can downstream AI models still figure out what was removed? Even if an object appears visually erased, the surrounding geometry, texture artifacts, and splat distributions often retain subtle, imperceptible cues that leak private semantic information.
Methods
The authors propose an automated, multi-faceted evaluation framework that measures object removal across three distinct granularities:
- Semantic Recognition ($\text{IoU}_{\text{drop}}$ & $\text{acc}_{\text{seg}}$): Evaluates semantic segmentation using GroundedSAM 2. $\text{IoU}_{\text{drop}}$ measures the performance drop of the detector on the target object after removal. $\text{acc}_{\text{seg}}$ tracks the percentage of post-removal rendered viewpoints where the object fails to be detected ($\text{IoU}_{\text{post}} < 0.5$).
- Instance-Level Trace Recognition ($\text{sim}_{\text{SAM}}$): Uses prompt-free Segment Anything (SAM) to generate instance masks across the entire frame. The similarity between the post-removal rendering masks and ground-truth post-removal masks is calculated via optimal 1-to-1 Hungarian matching. Successful removal reveals the background, which should closely match true post-removal instance layouts.
- Spatial/Geometric Modification ($\text{acc}_{\Delta\text{depth}}$): Computes rendered depth before and after removal. Using Generalized Histogram Thresholding to find a significant depth delta ($\xi_{\text{depth}}$), it computes the proportion of the target object's pixels whose depth actually changed, verifying that 3D spatial structure—not just surface color—was altered.
Datasets
- Remove360 (New Benchmark): 11 complex real-world scenes (5 indoor, 6 outdoor) captured with an Insta360 AcePro camera. Unlike existing single-object staged datasets (like 360-USID), Remove360 includes paired multi-view RGB captures taken physically before and after object removal (150–300 views each), paired with high-quality annotated ground-truth masks. Target removals range from compact items (backpacks, toy trucks) to complex, multi-instance objects (chairs, sofas, bicycles).
- Mip-NeRF 360: Evaluated on four standard scenes (counter, room, kitchen, garden) using pseudo-ground-truth masks generated with SAM and manual annotations.
Key Insights and Ideas
- Machine "Hallucinations" or Semantic Ghosts? Even when an object is completely invisible to human inspection in the edited render, GroundedSAM 2 can still classify and segment the removed object (e.g., backpacks or deck chairs). The residual pixel distribution mimics adversarial patterns, retaining class-identifying signals.
- Surface Erasure vs. Depth Failure: Many methods remove splat colors or foreground visibility but fail to update the underlying depth correctly ($\text{acc}_{\Delta\text{depth}}$ remains noticeably low in scenes like Living Room and Office).
- Current Methods Struggle with Real Complexity: Popular feature-distillation and projection methods that succeed on benchmark single objects (e.g., GaussianGrouping, Feature3DGS, SAGS) completely failed to remove objects in Remove360 scenes, leaving rendered scenes virtually unchanged because their segmentation front-ends failed under naturalistic multi-object clutter.
Key Results
The authors benchmarked five methods: Feature3DGS (FGS), GaussianGrouping (GG), SAGS, GaussianCut (GC), and AuraFusion360 (AF):
- Remove360 Performance: Only GC and AF successfully executed removals across scenes. Between them, GaussianCut achieved the strongest performance, particularly on mask matching ($\text{sim}_{\text{SAM}}$ up to 0.85 on Stroller and 0.83 on Chairs vs. AF's 0.72 and 0.62).
- Depth Gaps: In challenging indoor scenes like the Living Room (Sofa), both GC and AF only achieved an $\text{acc}_{\Delta\text{depth}}$ of 0.62, showing that a large fraction of object depth failed to adjust.
- Mip-NeRF 360 Comparison: On Mip-NeRF 360, AF, GC, and GG trade leads depending on the metric, with AF showing strong semantic drop ($\text{IoU}_{\text{drop}} = 0.95$ on Truck and $0.90$ on Table), while FGS and SAGS suffered from severe sensitivity to text prompts and high variance.
Limitations and Future Work
- Reliance on Pre-Trained Foundation Models: The semantic metrics rely on off-the-shelf vision models (SAM, GroundedSAM 2), which can introduce their own classification noise or misses.
- Inpainting Not Included: Evaluations were performed purely on removal without downstream multi-view background inpainting to isolate Gaussian editing behavior.
- Future Directions: The authors highlight the need for privacy-preserving deletion frameworks designed to leave zero latent artifacts, as well as studying whether adversarial attack algorithms could intentionally reconstruct private removed objects from these residual cues.
Summary updated