Remove360: Benchmarking Residuals After Object Removal in 3D Gaussian Splatting

Simona Kocour, Assia Benbihi, Torsten Sattler

TL;DR

Remove360 introduces a novel benchmark and evaluation protocol to detect "recoverable object evidence"—the semantic, structural, and geometric residuals left behind after deleting objects from 3D Gaussian Splatting scenes. The paper demonstrates that even when an object is visually erased to the human eye, downstream AI vision models can frequently still detect its prior existence through shadows, artifacts, or residual geometry. This work shifts the paradigm of 3D object removal from purely visual plausibility to a privacy-critical task of complete semantic erasure.

What Problem Does This Paper Address?

With the rapid proliferation of consumer-grade 3D scanning tools utilizing 3D Gaussian Splatting (3DGS), sharing realistic 3D scene reconstructions has become seamless. However, this raises significant privacy concerns: users often need to remove sensitive or private objects (like documents, photographs, or personal belongings) before sharing a scene.

Existing 3D object removal methods are typically evaluated based on visual plausibility or image-level reconstruction quality. The authors identify a critical flaw in this approach: a scene can look perfectly empty and plausible to a human observer while still containing subtle, "recoverable object evidence" (such as leftover shadows, boundary artifacts, or depth discontinuities). Downstream machine learning models can easily exploit these residuals to localize or identify the deleted object. Prior datasets were insufficient to study this because they were largely object-centric, staged, and lacked the cluttered environments and physical post-removal ground truth required for rigorous evaluation.

Fig. 1: Recoverable object evidence after object removal in 3D Gaussian Splatting. Although the target object is no longer apparent in the RGB rendering, our pipeline can still detect object-like evidence. Such evidence may arise from resid- ual geometry, boundary artifacts, shadows, reconstruction artifacts, or inpainting ar- tifacts. We evaluate removal quality using semantic, structural, and geometric probes. Top: scene before removal. Bottom: scene after removal. Left to right: RGB rendering, SAM [67] masks, GroundedSAM [36, 48, 70] overlay, and rendered depth.

Methods

The authors propose a multi-axis evaluation protocol that utilizes off-the-shelf foundation models as adversarial probes to test the completeness of object removal across three distinct dimensions:

  1. Semantic Suppression: Evaluates whether the category of the removed object remains detectable. An open-vocabulary segmentation model (GroundedSAM2) queries the edited scene using the target object's text prompt to compute the fraction of views where the object is successfully suppressed ($acc_{seg}$).
  2. Structural Change: Measures whether prompt-free, object-agnostic structures persist in the edited region. Segment Anything (SAM) masks are extracted before and after removal and matched via the Hungarian algorithm to compute structural similarity ($sim_{SAM}$).
  3. Geometric Modification: Assesses changes in the underlying 3D structure by calculating absolute differences in rendered depth maps ($acc_{\Delta depth}$) before and after editing, using Generalized Histogram Thresholding to automatically determine view-dependent change thresholds.

To unify these axes, the authors introduce the Removal Quality Score (RQS), computed as the geometric mean of the three axis scores. The geometric mean penalizes unbalanced performance, ensuring a method cannot achieve a high score by optimizing a single axis while failing on another. They also report Average Rank (AvgRank) across metrics as a scale-independent robustness check.

Datasets

The paper evaluates removal pipelines on two datasets:

  • Remove360 (New): A real-world benchmark featuring 11 cluttered indoor and outdoor scenes (e.g., Deckchair, Stroller, Sofa, Pillows). Each scene includes 150–300 views captured both before and after physically removing the target object, alongside manually refined object-level masks. This physical post-removal ground truth allows direct verification of whether the edited scene matches reality.
  • Mip-NeRF360: A standard dataset used to provide a complementary, reference-free evaluation setting (as it lacks physical post-removal captures).

Key Insights and Ideas

  • Visual Plausibility $\neq$ Semantic Erasure: A method can successfully trick a human eye while completely failing to fool a vision model. Leftover shadows or subtle geometric boundaries are highly informative to modern neural networks.
  • The Inpainting Trade-Off: Activating optional inpainting stages (such as in 3DGIC+i or AuraFusion360+i) consistently improves geometric modification scores by filling in depth. However, it frequently decreases structural agreement because the generative inpainting prior introduces new view-dependent or object-like artifacts.
  • 2D-First vs. 3D-First Pipelines: Removing objects from 2D training images before reconstructing the 3D scene (e.g., ObjectClearer) yields exceptional semantic suppression ($acc_{seg} = 0.976$) but struggles significantly to alter the underlying 3D geometry ($acc_{\Delta depth} = 0.320$), leaving flat, ghost-like geometric footprints.
Fig. 2: Semantic segmentation before and after removal on Remove360. GroundedSAM2 [36, 48, 70] detections shown before and after editing. Although the edited renderings appear visually plausible, the semantic probe [36, 48, 70] still iden- tifies target-associated evidence in the final representation. The predicted masks are used to compute IoUdrop (Eq. (1)) and accseg (Eq. (2)). Rows show different object removals.

Key Results

  • No Single Winner: No evaluated method dominated all three axes on the Remove360 dataset, highlighting distinct algorithmic trade-offs.
  • Top Performers: 3DGIC+i achieved the highest overall RQS (0.693) and the best average rank (2.33), closely followed by its removal-only variant, 3DGIC (0.690), due to their superior geometric modification capabilities.
  • Baseline Performance: The 2D-first baseline, ObjectClearer, dominated semantic suppression ($acc_{seg} = 0.976$) and structural agreement, but its low geometric score dragged its final RQS down to 0.588.
  • Fidelity vs. Residuals: Gaussian Cut achieved the highest pixel-level visual fidelity (PSNR of 18.28) against post-removal references but fell behind on the evidence-oriented axes.

Limitations and Future Work

  • Probe Dependence: The evaluation metrics act as adversarial probes using specific foundation models (SAM, GroundedSAM2). The scores are relative to these model families, and future work must validate how well these rankings generalize across different vision architectures.
  • Benchmark Scale: With 11 scenes, the benchmark successfully exposes massive per-axis performance gaps but lacks the statistical resolution to cleanly separate the top aggregate methods. Expanding the dataset with more scenes is noted as a priority.
  • Privacy Scope: The benchmark currently evaluates the detection of an object's existence and category. Protecting against human adversaries or preventing the recovery of high-fidelity pixel-level private data (like document text) remains outside the current scope.

Summary updated

Abstract

An object can disappear from a 3D scene, yet still be detectable. Even after visual removal, modern vision models may infer what was originally present. In this work, we introduce a novel benchmark and evaluation framework to quantify semantic residuals, the unintended cues left behind after object removal in 3D Gaussian Splatting. We conduct experiments across a diverse set of indoor and outdoor scenes, showing that current methods often preserve semantic information despite the absence of visual geometry. Notably, even when removal is followed by inpainting, residual cues frequently remain detectable by foundation models. We also present Remove360, a real-world dataset of pre- and post-removal RGB captures with object-level masks. Unlike prior datasets focused on isolated object instances, Remove360 contains complex, cluttered scenes that enable evaluation of object removal in full-scene settings. By leveraging the ground-truth post-removal images, we directly assess whether semantic presence is eliminated and whether downstream models can still infer what was removed. Our results reveal a consistent gap between geometric removal and semantic erasure, exposing critical limitations in existing 3D editing pipelines and highlighting the need for privacy-aware removal methods that eliminate recoverable cues, not only visible geometry. Dataset and evaluation code are publicly available.
BibTeX
@misc{kocour2025remove,
  author = {Simona Kocour and Assia Benbihi and Torsten Sattler},
  title = {{Remove360: Benchmarking Residuals After Object Removal in 3D Gaussian Splatting}},
  year = {2025},
  eprint = {2508.11431},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.11431},
}
Figure