In standard computer vision, we usually assume light travels in straight lines. When a photon leaves a source, it hits a surface and reflects directly into our camera. This "single-bounce" assumption is the bedrock of everything from traditional LiDAR to photogrammetry.
But the real world is often messy. In "participating media"—like thick clouds, dense fog, or murky water—photons don't just bounce once. They enter a volume and act like pinballs, scattering off millions of tiny particles before finally reaching a sensor. Traditionally, researchers treated this "multiple scattering" as noise to be filtered out.
A growing body of research, however, argues that this complexity is actually a signal. By modeling the physics of how light weaves through a volume, we can reconstruct 3D structures that were previously invisible.
From Noise to Signal
The fundamental shift in this field is moving beyond the single scattering assumption. In Structured Light in Scattering Media, Shree K. Nayar and his colleagues demonstrated that scattering isn't just a nuisance; it’s a source of geometric information.
When light scatters multiple times, it carries the "fingerprint" of the medium's density and microphysics. Aviad Levis and Yoav Y. Schechner have pioneered this "scattering-as-signal" approach in Multiple-Scattering Microphysics Tomography. Instead of trying to "see through" a cloud to a solid object, they use the multi-scattered light to map the cloud itself—recovering the 3D distribution of droplet sizes and densities.
The Physics of the Pinball Machine
To turn a blurry image of a cloud or a foggy street into a 3D model, we need a mathematical way to describe how light moves through a volume. This is governed by the Radiative Transfer Equation (RTE).
The RTE is notoriously difficult to solve because it is non-linear; every part of the volume affects every other part. In Airborne Three-Dimensional Cloud Tomography, Aviad Levis and his team tackled this by using "surrogate functions"—mathematical shortcuts that allow them to optimize the 3D density field without solving the full, grueling physics equations at every step. This allows for the reconstruction of heterogeneous volumes where light bounces repeatedly, a feat impossible with traditional straight-line tomography.
Differentiable Rendering: Backpropagating Through Fog
How do we actually "fit" a 3D model to a set of scattered images? The answer lies in differentiable rendering.
If we have a renderer that can simulate multiple scattering, and that renderer is differentiable, we can use gradient descent to tweak our 3D model until the simulated images match our real-world photos. Sebastian Weiss and Rüdiger Westermann introduced DiffDVR, a framework that makes this process memory-efficient. By analytically reversing the blending process, they allow researchers to optimize 3D density fields and lighting conditions simultaneously.
This approach is now being combined with generative AI. In Light Transport-aware Diffusion Posterior Sampling, researchers use a diffusion model as a "structural prior"—essentially teaching the system what a cloud should look like—while the differentiable physics engine ensures the 3D structure actually explains the light patterns seen in a single 2D photograph.
Real-Time Reconstruction with Gaussian Splatting
While early multi-scattering models took hours to converge, the field is moving toward real-time performance using 3D Gaussian Splatting (3DGS).
Standard 3DGS was designed for solid surfaces, but new variants are adapting it for volumes. Huapeng Li and Jonas Kulhanek developed WaterSplatting, which models the transition between air and water by fusing 3DGS with a dedicated volumetric field. Similarly, Terrence Chen introduced DDGS-CT, which disentangles isotropic and anisotropic light interactions to simulate X-ray scattering in real-time.
Why It Matters
The ability to decode multiple scattering has implications far beyond pretty pictures.
- Climate Science: Accurate 3D cloud tomography is essential for understanding how much solar energy the atmosphere absorbs versus reflects.
- Autonomous Systems: For self-driving cars or underwater drones, "seeing" through smoke or turbid water is a safety requirement. Shuhong Liu and Lin Gu have highlighted this in RealX3D, showing that current models still struggle to disentangle dense smoke from actual geometry.
- Medical Imaging: Techniques like Coded Aperture X-ray Scatter Imaging by Daniel L. Marks use scattered photons to reduce radiation doses while improving image quality.
By treating the "mess" of scattered light as a structured puzzle rather than random noise, these researchers are turning opaque environments into transparent ones.