Ray Deformation Networks for Novel View Synthesis of Refractive Objects

Weijian Deng, Dylan Campbell, Chunyi Sun, Shubham Kanitkar, Matthew E. Shaffer, Stephen Gould

Introduction

Neural Radiance Fields (NeRF) have revolutionized 3D scene reconstruction by enabling photorealistic novel view synthesis from sparse 2D images. However, a fundamental limitation of standard NeRF models is the assumption that light transport follows a straight path. This "straight-line rule" is violated in scenes containing refractive objects, such as glass containers or bodies of water, where light bends as it transitions between media.

Comparison of light transport assumptions: (a) standard opaque objects, (b) the challenge of refraction, and (c) the proposed ray deformation solution.

When standard NeRFs encounter refractive distortions, they often attempt to compensate by generating "ghost" geometry or noisy artifacts (often called "floaters") within the 3D volume. This results in blurry renderings and physically inconsistent reconstructions. While specialized methods exist for refraction, they typically require prior knowledge of object geometry or refractive indices. This paper introduces Ray Deformation Networks, a flexible framework designed to learn these complex light paths "in the wild."

Methodology

The proposed system integrates deformation networks directly into a NeRF pipeline (specifically the Nerfacto model). Instead of sampling points along a rigid linear ray, the model learns to warp the observation rays to match the physical reality of the scene.

Ray Deformation Field

The core of the approach consists of two small Multi-Layer Perceptrons (MLPs): a Position DeformNet ($\Psi_p$) and a Direction DeformNet ($\Psi_d$). These networks operate within a user-defined "deformable region," typically a bounding cuboid around the refractive object. For any point $x$ and direction $d$ sampled along a ray, the networks predict offsets ($\Delta x$ and $\Delta d$) that deform the ray's path dynamically.

Detailed view of the ray deformation mechanism, where the network predicts position and direction offsets to model curved light paths.

Regularization and Constraints

To prevent the networks from learning arbitrary or jagged paths, the authors introduce Collinearity Regularization. This loss term encourages piece-wise linearity, mimicking the behavior of light under Snell’s Law. Furthermore, a Near-Camera Density Penalty is employed to suppress the formation of artifacts near the camera lens, forcing the model to find a globally consistent 3D solution for the refractive interfaces. The model also predicts surface normals, supervised by the density field gradient, to better capture the geometry of refractive surfaces.

Experimental Results

The researchers evaluated Ray Deformation Networks across six challenging real-world datasets, including four new scenes featuring various liquids and containers ("Cup-A" through "D") and existing benchmarks ("Glass" and "Ball").

Quantitative Performance

The method consistently outperformed both general-purpose baselines and specialized refractive models. On the "Ball" dataset, the proposed method achieved a PSNR of 23.30, a significant improvement over Nerfacto (21.67) and Eikonal Fields (21.64). In case studies involving submerged objects (e.g., a pencil in water), the model outperformed competitors by over 1 dB.

Qualitative Quality

Visually, the reconstructions produced by Ray Deformation Networks are significantly cleaner. While baseline models often produce hollow or noisy interiors for glass objects, this approach reconstructs solid, well-defined geometry. The method effectively handles complex light paths and internal reflections that typically cause standard NeRFs to fail.

Conclusion

Ray Deformation Networks offer a robust solution for synthesizing novel views of refractive objects without requiring explicit physical parameters. By warping rays rather than the scene itself and applying physics-guided regularization, the model achieves state-of-the-art results on complex real-world data. Future work may focus on automating the localization of deformable regions and incorporating generative priors to recover high-frequency background details seen through refractive media.

Summary based on the research "Ray Deformation Networks for Novel View Synthesis of Refractive Objects".

Summary updated

Abstract

Neural Radiance Fields (NeRF) have demonstrated exceptional capabilities in creating photorealistic novel views using volume rendering on a radiance field. However, the intrinsic assumption of straight light rays within NeRF becomes a limitation when dealing with transparent or translucent objects that exhibit refraction, and therefore have curved light paths. This hampers the ability of these approaches to accurately model the appearance of refractive objects, resulting in suboptimal novel view synthesis and geometry estimates. To address this issue, we propose an innovative solution using deformable networks to learn a tailored deformation field for refractive objects. Our approach predicts position and direction offsets, allowing NeRF to model the curved light paths caused by refraction and therefore the complex and highly view-dependent appearances of refractive objects. We also introduce a regularization strategy that encourages piece-wise linear light paths, since most physical systems can be approximated with a piece-wise constant index of refraction. By seamlessly integrating our deformation networks into the NeRF framework, our method significantly improves rendering refractive objects from novel views.
BibTeX
@inproceedings{deng2024ray,
  author = {Weijian Deng and Dylan Campbell and Chunyi Sun and Shubham Kanitkar and Matthew E. Shaffer and Stephen Gould},
  title = {{Ray Deformation Networks for Novel View Synthesis of Refractive Objects}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00309},
}
Figure