TL;DR
Remove360 introduces a novel benchmark and evaluation protocol to detect "recoverable object evidence"—the semantic, structural, and geometric residuals left behind after deleting objects from 3D Gaussian Splatting scenes. The paper demonstrates that even when an object is visually erased to the human eye, downstream AI vision models can frequently still detect its prior existence through shadows, artifacts, or residual geometry. This work shifts the paradigm of 3D object removal from purely visual plausibility to a privacy-critical task of complete semantic erasure.
What Problem Does This Paper Address?
With the rapid proliferation of consumer-grade 3D scanning tools utilizing 3D Gaussian Splatting (3DGS), sharing realistic 3D scene reconstructions has become seamless. However, this raises significant privacy concerns: users often need to remove sensitive or private objects (like documents, photographs, or personal belongings) before sharing a scene.
Existing 3D object removal methods are typically evaluated based on visual plausibility or image-level reconstruction quality. The authors identify a critical flaw in this approach: a scene can look perfectly empty and plausible to a human observer while still containing subtle, "recoverable object evidence" (such as leftover shadows, boundary artifacts, or depth discontinuities). Downstream machine learning models can easily exploit these residuals to localize or identify the deleted object. Prior datasets were insufficient to study this because they were largely object-centric, staged, and lacked the cluttered environments and physical post-removal ground truth required for rigorous evaluation.
Methods
The authors propose a multi-axis evaluation protocol that utilizes off-the-shelf foundation models as adversarial probes to test the completeness of object removal across three distinct dimensions:
- Semantic Suppression: Evaluates whether the category of the removed object remains detectable. An open-vocabulary segmentation model (GroundedSAM2) queries the edited scene using the target object's text prompt to compute the fraction of views where the object is successfully suppressed ($acc_{seg}$).
- Structural Change: Measures whether prompt-free, object-agnostic structures persist in the edited region. Segment Anything (SAM) masks are extracted before and after removal and matched via the Hungarian algorithm to compute structural similarity ($sim_{SAM}$).
- Geometric Modification: Assesses changes in the underlying 3D structure by calculating absolute differences in rendered depth maps ($acc_{\Delta depth}$) before and after editing, using Generalized Histogram Thresholding to automatically determine view-dependent change thresholds.
To unify these axes, the authors introduce the Removal Quality Score (RQS), computed as the geometric mean of the three axis scores. The geometric mean penalizes unbalanced performance, ensuring a method cannot achieve a high score by optimizing a single axis while failing on another. They also report Average Rank (AvgRank) across metrics as a scale-independent robustness check.
Datasets
The paper evaluates removal pipelines on two datasets:
- Remove360 (New): A real-world benchmark featuring 11 cluttered indoor and outdoor scenes (e.g., Deckchair, Stroller, Sofa, Pillows). Each scene includes 150–300 views captured both before and after physically removing the target object, alongside manually refined object-level masks. This physical post-removal ground truth allows direct verification of whether the edited scene matches reality.
- Mip-NeRF360: A standard dataset used to provide a complementary, reference-free evaluation setting (as it lacks physical post-removal captures).
Key Insights and Ideas
- Visual Plausibility $\neq$ Semantic Erasure: A method can successfully trick a human eye while completely failing to fool a vision model. Leftover shadows or subtle geometric boundaries are highly informative to modern neural networks.
- The Inpainting Trade-Off: Activating optional inpainting stages (such as in 3DGIC+i or AuraFusion360+i) consistently improves geometric modification scores by filling in depth. However, it frequently decreases structural agreement because the generative inpainting prior introduces new view-dependent or object-like artifacts.
- 2D-First vs. 3D-First Pipelines: Removing objects from 2D training images before reconstructing the 3D scene (e.g., ObjectClearer) yields exceptional semantic suppression ($acc_{seg} = 0.976$) but struggles significantly to alter the underlying 3D geometry ($acc_{\Delta depth} = 0.320$), leaving flat, ghost-like geometric footprints.
Key Results
- No Single Winner: No evaluated method dominated all three axes on the Remove360 dataset, highlighting distinct algorithmic trade-offs.
- Top Performers: 3DGIC+i achieved the highest overall RQS (0.693) and the best average rank (2.33), closely followed by its removal-only variant, 3DGIC (0.690), due to their superior geometric modification capabilities.
- Baseline Performance: The 2D-first baseline, ObjectClearer, dominated semantic suppression ($acc_{seg} = 0.976$) and structural agreement, but its low geometric score dragged its final RQS down to 0.588.
- Fidelity vs. Residuals: Gaussian Cut achieved the highest pixel-level visual fidelity (PSNR of 18.28) against post-removal references but fell behind on the evidence-oriented axes.
Limitations and Future Work
- Probe Dependence: The evaluation metrics act as adversarial probes using specific foundation models (SAM, GroundedSAM2). The scores are relative to these model families, and future work must validate how well these rankings generalize across different vision architectures.
- Benchmark Scale: With 11 scenes, the benchmark successfully exposes massive per-axis performance gaps but lacks the statistical resolution to cleanly separate the top aggregate methods. Expanding the dataset with more scenes is noted as a priority.
- Privacy Scope: The benchmark currently evaluates the detection of an object's existence and category. Protecting against human adversaries or preventing the recovery of high-fidelity pixel-level private data (like document text) remains outside the current scope.
Summary updated