Differentiable Neural Surface Refinement for Modeling Transparent Objects

Weijian Deng, Dylan Campbell, Chunyi Sun, Shubham Kanitkar, Matthew E. Shaffer, Stephen Gould

Abstract

Reconstructing the 3D geometry of transparent objects remains a significant challenge for neural implicit surface methods due to the complex physics of light refraction and reflection. We present TNSR, a framework that incorporates differentiable physical ray tracing into neural surface reconstruction. By modeling Snell’s Law and Fresnel equations, TNSR refines surface estimates using background visual consistency as a supervisory signal. Our method achieves superior geometry estimation and visual fidelity on both synthetic and real-world datasets, even when the material's refractive index is unknown.

Introduction

The advent of Neural Radiance Fields (NeRF) has transformed 3D reconstruction, yet these models typically assume straight-line light propagation. This assumption fails for transparent media where light bends (refraction) and bounces (reflection). Existing methods for transparent objects often require controlled environments or prior knowledge of material properties. TNSR addresses these limitations by explicitly modeling the physics of light within a differentiable framework, enabling high-quality reconstruction from unconstrained multi-view RGB images.

Methodology

TNSR leverages the background environment as a source of supervision. The reconstruction pipeline begins with an initial surface estimate and a learned radiance field of the background.

  1. Physical Ray Tracing: Unlike standard methods, TNSR employs Snell’s Law to calculate ray bending at interfaces and Fresnel equations to determine the ratio of reflected to refracted light.
  2. Differentiable Refinement: The core innovation is a nested optimization approach for ray-surface intersections. This allows for analytical gradients that propagate color errors from the background back to the 3D surface parameters. If the color of the background seen through the object is incorrect, the model "nudges" the surface shape to correct the light path.
  3. Refractive Index Search: To handle unknown materials, the framework optimizes the refractive index by maximizing visual consistency across views.
Conceptual overview of ray-location correspondence and color consistency.

Key Results

TNSR demonstrates significant improvements in both geometric accuracy and rendering quality. On synthetic benchmarks like the "Bottle" dataset, TNSR achieved a lower Chamfer-L1 distance (29.54) compared to standard NeuS (31.05), indicating a more accurate 3D mesh. Qualitatively, the method eliminates common artifacts like "fogginess," producing sharp renders where optical distortions and reflections align perfectly with the real-world background.

Optimization of the refractive index value based on PSNR, showing the searched index closely matching the ground truth.

Conclusion

By integrating physical light transport into neural implicit surfaces, TNSR provides a robust solution for modeling transparent objects. While currently limited by the number of light bounces (two refractions and one reflection) and a lack of internal scattering support for translucent materials, the framework's use of analytical gradients and background supervision marks a major step toward general-purpose 3D reconstruction of complex optical materials.

Summary updated

Abstract

Neural implicit surface reconstruction leveraging volume rendering has led to significant advances in multi-view reconstruction. However, results for transparent objects can be very poor, primarily because the rendering function fails to account for the intricate light transport induced by refraction and reflection. In this study, we introduce trans-parent neural surface refinement (TNSR), a novel surface reconstruction framework that explicitly incorporates phys-ical refraction and reflection tracing. Beginning with an initial, approximate surface, our method employs sphere tracing combined with Snell's law to cast both reflected and refracted rays. Central to our proposal is an innovative differentiable technique devised to allow signals from the pho-tometric evidence to propagate back to the surface model by considering how the surface bends and reflects light rays. This allows us to connect surface refinement with volume rendering, enabling end-to-end optimization solely on multi-view RGB images. In our experiments, TNSR demonstrates significant improvements in novel view synthesis and geometry estimation of transparent objects, without prior knowledge of the refractive index.
BibTeX
@inproceedings{deng2024differentiable,
  author = {Weijian Deng and Dylan Campbell and Chunyi Sun and Shubham Kanitkar and Matthew E. Shaffer and Stephen Gould},
  title = {{Differentiable Neural Surface Refinement for Modeling Transparent Objects}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01916},
}
Figure