TRAN-D: 2D Gaussian Splatting-based Sparse-view Transparent Object Depth Reconstruction via Physics Simulation for Scene Update

Jeongyun Kim, Seunghoon Jeong, Giseop Kim, Myung-Hwan Jeon, Eunji Jun, Ayoung Kim

Abstract

Reconstructing the 3D geometry of transparent objects in sparse-view, dynamic scenes is a formidable challenge due to complex light transport and the need for frequent retraining. We introduce TRAN-D, a novel framework that leverages 2D Gaussian Splatting (2DGS) and physics-based simulation to achieve high-fidelity depth reconstruction. By incorporating an object-aware 3D loss and a Material Point Method (MPM) engine, TRAN-D can update scene changes—such as object removal or movement—in seconds using a single new image. Experimental results on synthetic and real-world datasets demonstrate that TRAN-D reduces depth error by over 39% compared to existing state-of-the-art methods while maintaining efficiency in sparse-view settings.

Introduction

The 3D reconstruction of transparent objects, such as glassware or plastic bottles, represents a "boss level" challenge in computer vision. Standard algorithms typically assume surfaces are opaque and diffuse; however, transparent materials reflect and refract light, violating these assumptions. While Neural Radiance Fields (NeRF) and Gaussian Splatting (GS) have shown promise, they often require dense viewpoints and extensive training. Furthermore, dynamic scenes pose a significant hurdle: if an object moves, most methods require a complete rescan and retraining of the entire model.

Comparison of TRAN-D against InstantSplat, highlighting the efficiency of physics-based scene updates over retraining.

TRAN-D addresses these limitations by focusing on two primary goals: accurate depth reconstruction from sparse views (3-6 images) and efficient model updates for dynamic scenes without starting from scratch.

Methodology

The TRAN-D pipeline consists of three integrated modules:

1. Segmentation Module

To isolate transparent objects from background clutter, the framework employs a fine-tuned Grounded SAM (Segment Anything Model). A unique "non-dictionary" text prompting strategy is used to uniquely identify and segment transparent objects, preventing interference from surrounding scene elements.

2. Object-aware 2D Gaussian Splatting (2DGS)

Unlike standard 3D Gaussian Splatting which uses volumetric "blobs," TRAN-D utilizes 2D planar disks. 2DGS is mathematically superior for representing thin, transparent surfaces. To ensure robustness in sparse-view scenarios, an Object-aware 3D Loss is introduced. This loss enforces uniform spacing among Gaussians and helps complete "obscured" regions that are not directly visible to the camera, preventing the model from overfitting to limited viewpoints.

3. Physics-based Scene Update

The most innovative component of TRAN-D is its handling of scene dynamics. When an object is removed or moved, the system converts the 2D Gaussians into a mesh and utilizes a Material Point Method (MPM) physics simulation (via the Taichi engine). This allows the system to predict how remaining objects will shift or topple due to gravity. A single subsequent "bird's-eye" image is then used to rapidly refine the new positions, reducing update times from minutes to approximately 14 seconds.

Key Insights

  • Surface-Centric Representation: Moving from 3DGS to 2DGS is critical for transparent objects, which are essentially "all surface." This representation is more geometrically faithful and stable during optimization.
  • Physics as a Prior: Rather than attempting to learn physics through data-heavy neural rendering, TRAN-D directly integrates a physics engine. This provides a strong geometric prior that accelerates the optimization of new scene states.
  • Efficiency: By using significantly fewer Gaussians (e.g., 33.5k vs. 850k in InstantSplat), TRAN-D is highly memory-efficient and faster to train.

Evaluation and Results

TRAN-D was evaluated on synthetic sequences from ClearPose and TRansPose, as well as real-world data captured using a Franka Emika Panda robot arm.

Qualitative comparison of depth reconstruction across various state-of-the-art methods. TRAN-D shows significantly clearer object boundaries and depth accuracy.

Key performance metrics include:

  • Accuracy: On the TRansPose dataset, TRAN-D achieved a 39% reduction in Mean Absolute Error (MAE) compared to baselines.
  • Sparse-view Stability: The model maintains high accuracy with as few as 3 training views, whereas competing methods like InstantSplat exhibit significant degradation.
  • Speed: Initial scene reconstruction takes less than a minute, and dynamic updates are completed in roughly 14 seconds.

Conclusion and Future Work

TRAN-D demonstrates a significant leap in handling transparent objects in dynamic, sparse-view environments. While the current framework relies heavily on initial segmentation quality and handles relatively simple dynamics, it sets a new benchmark for efficiency and accuracy. Future work will focus on making the pipeline segmentation-independent and extending the physics module to handle more complex interactions and fluid-like materials.

Summary updated

BibTeX
@inproceedings{kim2025tran,
  author = {Jeongyun Kim and Seunghoon Jeong and Giseop Kim and Myung-Hwan Jeon and Eunji Jun and Ayoung Kim},
  title = {{TRAN-D: 2D Gaussian Splatting-based Sparse-view Transparent Object Depth Reconstruction via Physics Simulation for Scene Update}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure