Remove360: Benchmarking Residuals After Object Removal in 3D Gaussian Splatting

Simona Kocour, Assia Benbihi, Torsten SattlerSubmission to ICLR 2026 · Withdrawn

TL;DR

When you delete a sensitive object from a 3D scene, is it truly gone? This paper introduces Remove360, a real-world benchmark and four-part evaluation suite that reveals how state-of-the-art 3D Gaussian Splatting (3DGS) removal techniques often leave behind invisible "semantic residuals"—traces that humans miss, but modern vision foundation models can still detect and reconstruct.

What Problem Does This Paper Address?

As casual 3D capture apps become commonplace on smartphones and AR headsets, users increasingly need to erase private items—such as family photos, personal documents, or specific indoor decor—before publishing or sharing 3D reconstructions.

While 3D Gaussian Splatting (3DGS) makes deletion conceptually easy (simply prune the Gaussians corresponding to the target), current literature evaluates removal mostly through visual aesthetics or foreground segmentation. It ignores a critical privacy question: Can downstream AI models still figure out what was removed? Even if an object appears visually erased, the surrounding geometry, texture artifacts, and splat distributions often retain subtle, imperceptible cues that leak private semantic information.

Methods

The authors propose an automated, multi-faceted evaluation framework that measures object removal across three distinct granularities:

  1. Semantic Recognition ($\text{IoU}_{\text{drop}}$ & $\text{acc}_{\text{seg}}$): Evaluates semantic segmentation using GroundedSAM 2. $\text{IoU}_{\text{drop}}$ measures the performance drop of the detector on the target object after removal. $\text{acc}_{\text{seg}}$ tracks the percentage of post-removal rendered viewpoints where the object fails to be detected ($\text{IoU}_{\text{post}} < 0.5$).
  2. Instance-Level Trace Recognition ($\text{sim}_{\text{SAM}}$): Uses prompt-free Segment Anything (SAM) to generate instance masks across the entire frame. The similarity between the post-removal rendering masks and ground-truth post-removal masks is calculated via optimal 1-to-1 Hungarian matching. Successful removal reveals the background, which should closely match true post-removal instance layouts.
  3. Spatial/Geometric Modification ($\text{acc}_{\Delta\text{depth}}$): Computes rendered depth before and after removal. Using Generalized Histogram Thresholding to find a significant depth delta ($\xi_{\text{depth}}$), it computes the proportion of the target object's pixels whose depth actually changed, verifying that 3D spatial structure—not just surface color—was altered.

Datasets

  • Remove360 (New Benchmark): 11 complex real-world scenes (5 indoor, 6 outdoor) captured with an Insta360 AcePro camera. Unlike existing single-object staged datasets (like 360-USID), Remove360 includes paired multi-view RGB captures taken physically before and after object removal (150–300 views each), paired with high-quality annotated ground-truth masks. Target removals range from compact items (backpacks, toy trucks) to complex, multi-instance objects (chairs, sofas, bicycles).
  • Mip-NeRF 360: Evaluated on four standard scenes (counter, room, kitchen, garden) using pseudo-ground-truth masks generated with SAM and manual annotations.

Key Insights and Ideas

  • Machine "Hallucinations" or Semantic Ghosts? Even when an object is completely invisible to human inspection in the edited render, GroundedSAM 2 can still classify and segment the removed object (e.g., backpacks or deck chairs). The residual pixel distribution mimics adversarial patterns, retaining class-identifying signals.
  • Surface Erasure vs. Depth Failure: Many methods remove splat colors or foreground visibility but fail to update the underlying depth correctly ($\text{acc}_{\Delta\text{depth}}$ remains noticeably low in scenes like Living Room and Office).
  • Current Methods Struggle with Real Complexity: Popular feature-distillation and projection methods that succeed on benchmark single objects (e.g., GaussianGrouping, Feature3DGS, SAGS) completely failed to remove objects in Remove360 scenes, leaving rendered scenes virtually unchanged because their segmentation front-ends failed under naturalistic multi-object clutter.

Key Results

The authors benchmarked five methods: Feature3DGS (FGS), GaussianGrouping (GG), SAGS, GaussianCut (GC), and AuraFusion360 (AF):

  • Remove360 Performance: Only GC and AF successfully executed removals across scenes. Between them, GaussianCut achieved the strongest performance, particularly on mask matching ($\text{sim}_{\text{SAM}}$ up to 0.85 on Stroller and 0.83 on Chairs vs. AF's 0.72 and 0.62).
  • Depth Gaps: In challenging indoor scenes like the Living Room (Sofa), both GC and AF only achieved an $\text{acc}_{\Delta\text{depth}}$ of 0.62, showing that a large fraction of object depth failed to adjust.
  • Mip-NeRF 360 Comparison: On Mip-NeRF 360, AF, GC, and GG trade leads depending on the metric, with AF showing strong semantic drop ($\text{IoU}_{\text{drop}} = 0.95$ on Truck and $0.90$ on Table), while FGS and SAGS suffered from severe sensitivity to text prompts and high variance.

Limitations and Future Work

  • Reliance on Pre-Trained Foundation Models: The semantic metrics rely on off-the-shelf vision models (SAM, GroundedSAM 2), which can introduce their own classification noise or misses.
  • Inpainting Not Included: Evaluations were performed purely on removal without downstream multi-view background inpainting to isolate Gaussian editing behavior.
  • Future Directions: The authors highlight the need for privacy-preserving deletion frameworks designed to leave zero latent artifacts, as well as studying whether adversarial attack algorithms could intentionally reconstruct private removed objects from these residual cues.

Summary updated

Abstract

Understanding what semantic information persists after object removal is critical for privacy-preserving 3D reconstruction and editable scene representations. In this work, we introduce a novel benchmark and evaluation framework to measure semantic residuals—the unintended semantic traces left behind—after object removal in 3D Gaussian Splatting. We conduct experiments across a diverse set of indoor and outdoor scenes, showing that current methods can preserve semantic information despite the absence of visual geometry. We also release Remove360, a dataset of pre/post-removal RGB images and object-level masks captured in real-world environments. While prior datasets have focused on isolated object instances, Remove360 covers a broader and more complex range of indoor and outdoor scenes, enabling evaluation of object removal in the context of full-scene representations. Given ground truth images of a scene before and after object removal, we assess whether we can truly eliminate semantic presence, and if downstream models can still infer what was removed. Our findings reveal critical limitations in current 3D object removal techniques and underscore the need for more robust solutions capable of handling real-world complexity.
BibTeX
@misc{kocour2025remove,
  author = {Simona Kocour and Assia Benbihi and Torsten Sattler},
  title = {{Remove360: Benchmarking Residuals After Object Removal in 3D Gaussian Splatting}},
  year = {2025},
  eprint = {2508.11431},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.11431},
}
Figure