3D Reconstruction Through Refractive Interfaces

146 concepts69 papers1 proposal

Most computer vision algorithms share a foundational assumption: light travels in straight lines. From traditional Structure-from-Motion (SfM) to modern Neural Radiance Fields (NeRFs), the geometry of the visual world is almost universally mapped using the pinhole camera model. In this model, every pixel on an image sensor corresponds to a straight line-of-sight ray, and all rays intersect at a single point in space—the optical center.

However, when a camera images a scene through a transparent medium like water, glass, or a containment vessel, this assumption breaks down entirely. As light passes from one medium to another, its speed changes, causing the ray to bend at the boundary. This phenomenon, known as refraction, violates the single-viewpoint assumption and introduces severe, depth-dependent geometric distortions that traditional camera models cannot resolve.

The Breakdown of the Single Viewpoint

To understand why standard computer vision fails in refractive environments, consider a camera looking through a flat glass window into an aquarium. As a light ray travels from a submerged object through the water, passes through the glass, and enters the air toward the camera lens, it bends twice according to Snell's Law.

If we back-project the rays from the camera sensor into the water, a striking geometric problem emerges: the rays no longer intersect at a single point. Instead of a single optical center, refraction creates a locus of virtual viewpoints called a caustic.

Because there is no single point of projection, the distortion is fundamentally scene-dependent and depth-dependent. As noted by Chang et al., this means refraction cannot be treated as a static, two-dimensional lens distortion (like radial distortion). If an object moves closer to or further from the refractive interface, the angle of incidence changes, altering the path of the light non-linearly.

When standard pinhole models are forced to interpret these bent rays, they introduce systematic errors. As demonstrated by the authors of Refractive Structure-from-Motion on Underwater Images, ignoring this double refraction causes estimated camera trajectories to physically drift and warps the reconstructed 3D geometry of the scene. Flat surfaces are incorrectly reconstructed as curved or slanted structures because the algorithm misinterprets the non-linear displacement of light as physical depth variations.

Standard Pinhole Model (Straight Rays)
Camera Center (•) ──────────────────────────────> Object Point

Refractive Model (Bent Rays)
Camera Center (•) ─── Air ───> | Glass | ─── Water ───> Submerged Object
                               Interface

Physics-Based Ray Tracing

The most direct way to restore geometric fidelity is to replace the straight-line assumption with physics-based ray tracing. By explicitly modeling the geometry and refractive indices of the intervening media, algorithms can mathematically "undo" the bending of light.

Yamashita demonstrated this by developing a multi-media ray tracing framework for 3D scanning. By applying Snell's Law at every boundary (air-to-glass and glass-to-liquid), the system traces the exact non-linear path of both the camera's observation rays and a laser projector's illumination rays. This explicit modeling reduces measurement errors from several millimeters down to sub-millimeter precision, preserving sharp geometric features like edges that would otherwise be blurred or warped by uncorrected refraction.

To perform 3D triangulation in these environments, researchers use a backward projection strategy. For instance, Lin Gui et al. proposed casting rays from the camera center through individual pixels and calculating their refraction vectors at each medium transition. By leveraging the principle of light reversibility, the 3D coordinates of a point can be determined by finding the point of minimum distance between refracted rays originating from different camera views. This approach bypasses the complex, non-linear mathematics required to map a 3D point back onto a 2D sensor plane.

Linearizing the Refractive Camera

While explicit ray tracing is highly accurate, solving the forward projection problem—determining exactly where a known 3D point will project onto the camera sensor through multiple refractive layers—typically requires solving computationally expensive high-degree polynomials.

To overcome this bottleneck, Kawahara et al. introduced a "pixel-wise varifocal" camera model. Instead of treating the camera as a single system with a fixed focal length, they assigned a unique virtual focal length to every individual pixel based on the geometry of the flat housing. This parameterization transforms a complex 12th-degree polynomial optimization problem into an efficient, linear framework, enabling fast extrinsic calibration of underwater cameras without sacrificing physical accuracy.

Similarly, incorporating these physical constraints directly into bundle adjustment algorithms allows for the simultaneous estimation of camera poses and interface parameters. Haner et al. and Chadebecq demonstrated that embedding the geometry of flat refractive interfaces into the structure-from-motion pipeline dramatically minimizes reprojection errors. This physics-grounded optimization prevents the shape warping and trajectory drift common in small-baseline applications like medical endoscopy and deep-sea mapping.

Taming Dynamic and Wavy Interfaces

The problem becomes significantly more challenging when the refractive interface is not a flat piece of glass, but a dynamic, turbulent fluid surface, such as wavy water. A moving water surface acts as a collection of constantly changing, non-planar lenses that distort, shift, and split the light rays passing through them.

To reconstruct scenes through turbulent boundaries, algorithms must solve a dual problem: they must map the unknown, time-varying geometry of the fluid surface while simultaneously resolving the structure of the underlying scene. Yiming Qian et al. addressed this by optimizing for surface normals and scene points jointly. Crucially, the authors avoided enforcing spatial smoothness constraints on the submerged scene. Because refraction non-linearly maps light paths, adjacent pixels on a camera sensor can be bent to look at widely separated points in the underwater environment. Treating the submerged scene as independent spatial points rather than a continuous manifold allows the system to handle these sharp discontinuities accurately.

Wavy Water Surface (Dynamic Lenses)
   ~ ~ \_ _/ ~ ~ \__/~ ~
      /     \       \
     /       \       \   <-- Rays bend unpredictably
    •         •       •
   Adjacent pixels map to distant object points

With the advent of neural rendering, this physics-based approach has been integrated into implicit volumetric representations. Traditional Neural Radiance Fields (NeRFs) fail when rendering through water because they assume straight-line ray marching. Zhan et al. introduced NeRFrac, a framework that models the boundary as a dedicated neural refractive field. By using the vector form of Snell's Law to bend the camera rays at the estimated interface before marching through the volume, the model achieves highly accurate view synthesis.

A compelling advantage of this physics-grounded neural approach is the ability to perform digital restoration. During inference, researchers can simulate straight-line light transport, effectively "removing" the water layer entirely to render a perfectly clear, undistorted view of an object as if it were sitting in a vacuum.

Key Foundational Work

3-D Shape Measurement of Underwater Object by Temporal-Spatial Image Analysis (1996) — contributed 5 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 71 papers
  2. Version 1 · 2026-10-04 · 55 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_refractive_distortion_6771ffc5,
  title = {refractive distortion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/refractive-distortion/rdr_6771ffc5355a4358b9b38d2aa65f5db4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_refractive_distortion_6771ffc5_v2,
  title = {refractive distortion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/refractive-distortion/rdr_6771ffc5355a4358b9b38d2aa65f5db4/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake