Is there anything left? Measuring semantic residuals of objects removed from 3D Gaussian Splatting

Simona Kocour, Assia Benbihi, Aikaterini Adam, Torsten SattlerarXiv 2025 · DOI · arXiv

Introduction

As 3D Gaussian Splatting (3DGS) becomes a standard for high-fidelity scene reconstruction, the ability to edit these scenes—specifically removing objects—has gained significant attention. However, current research focuses almost exclusively on the visual quality of the removal or the accuracy of the initial segmentation. This paper investigates the "semantic residual": the data that remains after an object is supposedly deleted, which may still be interpretable by machine learning models even if invisible to the human eye. This phenomenon poses a significant privacy risk, as sensitive information could be recovered from "cleaned" scenes by automated systems.

The Challenge of Semantic Residuals

The core problem identified is that existing removal methods often leave behind "ghosts"—subtle feature patterns or geometric artifacts that act like adversarial residuals. While a human viewer might see a clean floor where a table once stood, a semantic segmentation model might still identify the table's presence with high confidence based on lingering high-dimensional features.

Visualizing semantic residuals across different modalities: RGB renders, SAM segments, semantic segmentation, and depth maps before and after object removal.

Evaluation Framework: Three Pillars of Removal

To quantify these residuals, the authors propose a multi-modal evaluation framework consisting of three distinct metrics:

  1. Semantic Object Recognition ($IoU_{drop}$): This measures the change in a model's ability to segment the target object before and after removal. A high drop indicates that the semantic features associated with that object class have been successfully purged.
  2. Anything Recognition ($sim_{SAM}$): Utilizing the Segment Anything Model (SAM), this metric compares mask distributions. If an object is truly removed, SAM should naturally segment the background behind it. If the masks remain largely unchanged, the "ghost" of the object is likely still influencing the model's perception.
  3. Spatial Recognition ($acc_{\Delta depth}$): This measures physical changes in the scene geometry. For instance, removing a table should result in a depth map that reflects the distance to the floor or wall behind it, rather than the original table surface.

Graph-Based Refinement

Beyond evaluation, the paper introduces a Graph-Based Refinement method to improve removal quality. This optimization step takes the output of existing tools and "cleans" the scene by enforcing spatial and semantic consistency. By treating the Gaussians as nodes in a graph, the method can more effectively prune stray elements that share the same space or semantic features as the target object, preventing the leakage of private information through invisible traces.

Results and Key Insights

The study utilized the mipNeRF360 dataset, employing pseudo-ground-truth masks for rigorous testing. The findings highlight several critical points:

  • Top Performers: GaussianCut (GC) and GaussianGrouping (GG) were the most effective at minimizing residuals, largely due to their use of 3D spatial regularization.
  • The Perception Gap: One of the most striking results is the confirmation that objects can be invisible to humans while remaining clear to neural networks. As shown in the comparison of SAM segments, even after removal, the model often retains a contour of the deleted object.
Comparison of SAM segments before and after object removal using Gaussian Grouping, highlighting the persistence of semantic contours.
  • Metric Complementarity: No single metric is sufficient. A scene might pass a depth check but fail a semantic check due to "texture ghosts" left on the surrounding geometry. A comprehensive evaluation requires looking at semantics, distribution, and geometry simultaneously.

Conclusion

This research shifts the focus of 3D scene editing from purely aesthetic results to a more rigorous, privacy-conscious standard. By identifying and measuring semantic residuals, the authors provide a roadmap for developing "residual-aware" 3D representations. Future work should aim to integrate these metrics directly into the training and editing pipelines of 3DGS models to ensure that when an object is removed, it is truly gone.

Summary updated

Abstract

Searching in and editing 3D scenes has become extremely intuitive with trainable scene representations that allow linking human concepts to elements in the scene. These operations are often evaluated on the basis of how accurately the searched element is segmented or extracted from the scene. In this paper, we address the inverse problem, that is, how much of the searched element remains in the scene after it is removed. This question is particularly important in the context of privacy-preserving mapping when a user reconstructs a 3D scene and wants to remove private elements before sharing the map. To the best of our knowledge, this is the first work to address this question. To answer this, we propose a quantitative evaluation that measures whether a removal operation leaves object residuals that can be reasoned over. The scene is not private when such residuals are present. Experiments on state-of-the-art scene representations show that the proposed metrics are meaningful and consistent with the user study that we also present. We also propose a method to refine the removal based on spatial and semantic consistency.
BibTeX
@misc{kocour2025is,
  author = {Simona Kocour and Assia Benbihi and Aikaterini Adam and Torsten Sattler},
  title = {{Is there anything left? Measuring semantic residuals of objects removed from 3D Gaussian Splatting}},
  year = {2025},
  eprint = {2503.17574},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.17574},
}
Figure