2D Gaussian Splatting-based Sparse-view Transparent Object Depth Reconstruction via Physics Simulation for Scene Update

Jeongyun Kim, Seunghoon Jeong, Giseop Kim, Myung-Hwan Jeon, Eunji Jun, Ayoung Kim

TL;DR

This paper introduces TRAN-D, a novel method for reconstructing the 3D depth of transparent objects from sparse views. It leverages 2D Gaussian Splatting, object-aware segmentation, and physics simulation to achieve accurate and efficient reconstruction, even in dynamic scenes with object removal.

What Problem Does This Paper Address?

Reconstructing the 3D geometry of transparent objects is notoriously difficult due to their inherent properties like reflection and refraction. Existing methods often struggle with sparse input views and dynamic scenes where objects move or are removed, requiring extensive training or complete rescans. While recent advancements in neural rendering like Gaussian Splatting (GS) have shown promise, they often fail to accurately handle transparent objects, misinterpreting boundaries and leading to artifacts. This paper aims to bridge this gap by providing a robust and efficient method for transparent object depth reconstruction in challenging sparse-view and dynamic scenarios.

Methods

TRAN-D is built upon three core modules:

  • Segmentation Module: This module utilizes a fine-tuned Grounded SAM (Segment Anything Model) to isolate transparent objects from the background. By using category-specific prompts, it ensures consistent instance segmentation across multiple views, which is crucial for focusing the reconstruction process.
  • Object-aware 2D Gaussian Splatting (2DGS): Instead of traditional 3DGS, TRAN-D employs 2DGS, which is better suited for surface modeling. The key innovation here is an "object-aware 3D loss." This loss function strategically guides the placement of Gaussians, especially in obscured regions, by enforcing uniform spacing and density. This prevents floaters and ensures a more complete and accurate surface representation. The method also renders and optimizes object index one-hot vectors to clearly delineate between multiple transparent objects.
  • Physics Simulation for Scene Update: To handle dynamic scenes (e.g., object removal), TRAN-D integrates a physics-based simulation. After an object is removed, the system uses a generated mesh from the prior reconstruction to simulate the resulting chain-reaction movements of other objects. This simulation then informs a re-optimization of the Gaussian representation, allowing the scene to be updated efficiently without requiring a full rescan.

Datasets

The paper uses synthetic datasets for evaluation:

  • TransPose: Used for evaluating depth reconstruction in sparse-view scenarios.
  • ClearPose: Another synthetic dataset used for depth reconstruction evaluation.

The authors also created their own synthetic sequences with various backgrounds and textures for quantitative evaluation, specifically designed to include object removal scenarios, as no existing benchmarks covered this. For real-world experiments, they captured their own sequences using a Franka Emika Panda arm and a RealSense L515 camera.

Key Insights and Ideas

  • 2DGS for Transparent Objects: The paper highlights that 2D Gaussian Splatting, with its focus on surface modeling, is more effective for transparent objects than 3DGS, which can struggle with multi-view inconsistencies.
  • Object-Awareness is Key: By explicitly segmenting transparent objects and using object-specific losses, the method can overcome the inherent challenges of their reflective and refractive properties, leading to significantly improved depth accuracy.
  • Physics Simulation for Dynamics: Integrating physics simulation is a crucial step for handling scene changes like object removal. It allows for a seamless update of the scene representation, avoiding the need for complete rescans and maintaining reconstruction quality.
  • Sparse Views are Sufficient: The method demonstrates robust performance even with a single image for scene updates at $t=1$, showcasing the power of the physics simulation and object-aware optimization in sparse settings.

Key Results

  • Depth Accuracy: TRAN-D significantly outperforms existing methods on both synthetic datasets. For example, on the TransPose sequences, it reduces the Mean Absolute Error (MAE) by over 39%. It also achieves a $\delta < 2.5$ cm accuracy of 48.46% using just one image for update, which is 1.5 times better than baselines using six images.
  • Efficiency: TRAN-D is highly efficient. The initial training takes under a minute, and scene updates at $t=1$ require only about 13.8 seconds. This is attributed to the reduced number of Gaussians needed due to object-specific optimization and the object-aware loss.
  • Qualitative Results: The paper shows that TRAN-D produces dense and artifact-free depth reconstructions, even capturing fine details like cup handles, which are missed by other methods. The physics simulation effectively handles object removal and subsequent scene adjustments.

Limitations and Future Work

  • Segmentation Dependency: The performance of TRAN-D is heavily reliant on the quality of the segmentation. Failures in segmentation due to tracking issues, intense lighting, or difficult backgrounds can degrade performance.
  • Limited Dynamics Handling: Currently, the method primarily handles partial object removal or slight movements.
  • Future Directions: The authors suggest developing more robust, segmentation-independent approaches that can handle more complex dynamics and lighting environments. This would broaden the applicability of their method to a wider range of real-world scenarios.

Summary updated

Abstract

Understanding the 3D geometry of transparent objects from RGB images is challenging due to their inherent physical properties, such as reflection and refraction. To address these difficulties, especially in scenarios with sparse views and dynamic environments, we introduce TRAN-D, a novel 2D Gaussian Splatting-based depth reconstruction method for transparent objects. Our key insight lies in separating transparent objects from the background, enabling focused optimization of Gaussians corresponding to the object. We mitigate artifacts with an object-aware loss that places Gaussians in obscured regions, ensuring coverage of invisible surfaces while reducing overfitting. Furthermore, we incorporate a physics-based simulation that refines the reconstruction in just a few seconds, effectively handling object removal and chain-reaction movement of remaining objects without the need for rescanning. TRAN-D is evaluated on both synthetic and real-world sequences, and it consistently demonstrated robust improvements over existing GS-based state-of-the-art methods. In comparison with baselines, TRAN-D reduces the mean absolute error by over 39% for the synthetic TRansPose sequences. Furthermore, despite being updated using only one image, TRAN-D reaches a δ < 2.5 cm accuracy of 48.46%, over 1.5 times that of baselines, which uses six images. Code and more results are available at https://jeongyun0609.github.io/TRAN-D/.
BibTeX
@inproceedings{kim2025d,
  author = {Jeongyun Kim and Seunghoon Jeong and Giseop Kim and Myung-Hwan Jeon and Eunji Jun and Ayoung Kim},
  title = {{2D Gaussian Splatting-based Sparse-view Transparent Object Depth Reconstruction via Physics Simulation for Scene Update}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure