Introduction
Neural Radiance Fields (NeRF) have revolutionized 3D scene reconstruction by enabling photorealistic novel view synthesis from sparse 2D images. However, a fundamental limitation of standard NeRF models is the assumption that light transport follows a straight path. This "straight-line rule" is violated in scenes containing refractive objects, such as glass containers or bodies of water, where light bends as it transitions between media.
When standard NeRFs encounter refractive distortions, they often attempt to compensate by generating "ghost" geometry or noisy artifacts (often called "floaters") within the 3D volume. This results in blurry renderings and physically inconsistent reconstructions. While specialized methods exist for refraction, they typically require prior knowledge of object geometry or refractive indices. This paper introduces Ray Deformation Networks, a flexible framework designed to learn these complex light paths "in the wild."
Methodology
The proposed system integrates deformation networks directly into a NeRF pipeline (specifically the Nerfacto model). Instead of sampling points along a rigid linear ray, the model learns to warp the observation rays to match the physical reality of the scene.
Ray Deformation Field
The core of the approach consists of two small Multi-Layer Perceptrons (MLPs): a Position DeformNet ($\Psi_p$) and a Direction DeformNet ($\Psi_d$). These networks operate within a user-defined "deformable region," typically a bounding cuboid around the refractive object. For any point $x$ and direction $d$ sampled along a ray, the networks predict offsets ($\Delta x$ and $\Delta d$) that deform the ray's path dynamically.
Regularization and Constraints
To prevent the networks from learning arbitrary or jagged paths, the authors introduce Collinearity Regularization. This loss term encourages piece-wise linearity, mimicking the behavior of light under Snell’s Law. Furthermore, a Near-Camera Density Penalty is employed to suppress the formation of artifacts near the camera lens, forcing the model to find a globally consistent 3D solution for the refractive interfaces. The model also predicts surface normals, supervised by the density field gradient, to better capture the geometry of refractive surfaces.
Experimental Results
The researchers evaluated Ray Deformation Networks across six challenging real-world datasets, including four new scenes featuring various liquids and containers ("Cup-A" through "D") and existing benchmarks ("Glass" and "Ball").
Quantitative Performance
The method consistently outperformed both general-purpose baselines and specialized refractive models. On the "Ball" dataset, the proposed method achieved a PSNR of 23.30, a significant improvement over Nerfacto (21.67) and Eikonal Fields (21.64). In case studies involving submerged objects (e.g., a pencil in water), the model outperformed competitors by over 1 dB.
Qualitative Quality
Visually, the reconstructions produced by Ray Deformation Networks are significantly cleaner. While baseline models often produce hollow or noisy interiors for glass objects, this approach reconstructs solid, well-defined geometry. The method effectively handles complex light paths and internal reflections that typically cause standard NeRFs to fail.
Conclusion
Ray Deformation Networks offer a robust solution for synthesizing novel views of refractive objects without requiring explicit physical parameters. By warping rays rather than the scene itself and applying physics-guided regularization, the model achieves state-of-the-art results on complex real-world data. Future work may focus on automating the localization of deformable regions and incorporating generative priors to recover high-frequency background details seen through refractive media.
Summary based on the research "Ray Deformation Networks for Novel View Synthesis of Refractive Objects".
Summary updated