Differentiable Path Tracing for Inverse Rendering

This direction is split into other directions: differentiable ray tracing. Its last version stays readable and citable here.

Version 1 of 197 papers119 conceptsfrozen 2026-10-04

In computer graphics, the "forward" problem is well-understood: given a 3D scene, a camera, and light sources, how do we generate a realistic image? The "inverse" problem is much harder: given one or more images, how do we recover the underlying geometry, materials, and lighting?

Historically, researchers used "black-box" neural networks to guess these parameters or relied on simplified shading models (like Lambertian reflectance) that ignore the complex ways light actually behaves. However, a new direction has emerged that integrates the fundamental physics of light transport directly into the optimization loop. By making the entire path-tracing process differentiable, we can treat the laws of physics as a bridge, allowing gradients to flow from the pixels of a photograph back to the physical properties of the scene.

The Physics of the Gradient

At the heart of this research is the realization that light transport is a mathematical function that can be differentiated. In a standard path tracer, we simulate light by tracing rays from a camera, bouncing them off surfaces, and calculating how much energy reaches the sensor. To perform inverse rendering, we need to know how a small change in a surface's roughness or a light's position would change the final pixel value.

Dejan Azinovic and Tzu-Mao Li demonstrated in Inverse Path Tracing that by using a differentiable Monte Carlo path tracer, we can jointly estimate materials and lighting with far greater accuracy than traditional methods. Instead of "baking" shadows and reflections into a texture, the system understands that a dark patch on a wall is a shadow caused by an occluder, not a dark stain on the wallpaper.

This physical grounding is crucial because it ensures the recovered assets are "relightable." If you move a light source in a digitally reconstructed room, the shadows and reflections should move naturally. As Cheng Sun shows in Neural-PBIR, simulating multi-bounce global illumination allows the model to decouple material properties from environmental lighting, preventing the "baked-in" artifacts that plague simpler neural approximations.

Differentiating Through Discontinuities

One of the greatest challenges in this field is that light transport is full of "discontinuities." A ray either hits an object or it doesn't; a surface is either in shadow or in light. These hard edges create "non-differentiable" steps that break standard gradient descent.

To solve this, researchers like Nicolas Roussel and Wenzel Jakob developed Many-Worlds Inverse Rendering. They proposed a unified transport law that handles shading and boundary derivatives together. By treating potential surface patches as mutually exclusive "worlds," they created a Monte Carlo algorithm that can differentiate through complex light paths—like reflections seen through a curved glass—without needing specialized silhouette-tracing heuristics.

Similarly, Tristan Aumentado-Armstrong and Allan D. Jepson introduced Directed Distance Fields, which treat neural network queries as recursive ray bounces. This allows the system to model global illumination within an implicit framework, using the output of one ray-surface intersection as the input for the next bounce.

Beyond Pixels: Sensors and Tomography

The power of differentiable light transport extends beyond standard RGB images. If we can model how light moves, we can model almost any optical sensor.

  • Refractive Tomography: Brandon Zhao and Pratul P. Srinivasan used Hamilton’s equations within a differentiable pipeline to reconstruct the 3D structure of transparent, light-bending media from a single view in Single View Refractive Index Tomography.
  • Depth Sensors: Benjamin Planche developed DDS, which makes the entire process of a structured-light sensor (like a Kinect) differentiable, from the projected pattern to the internal stereo-matching logic.
  • LiDAR: In LiDAR-RT, Chenxu Zhou integrated hardware-accelerated ray tracing with 3D Gaussian Splatting to simulate how laser pulses reflect off dynamic driving scenes in real-time.

Integrating with Gaussian Splatting

The recent explosion of 3D Gaussian Splatting (3DGS) has provided a new arena for these ideas. While original 3DGS was a fast rasterization technique, it lacked physical light transport. Researchers are now "upgrading" it with ray-tracing capabilities.

H. Chen and Zehong Lin introduced GI-GS, which adds a differentiable path-tracing pass to Gaussian Splatting. By tracing rays from the surface of the Gaussians to evaluate visibility and secondary bounces, they can accurately decompose scenes into geometry and materials. Chun Gu pushed this further with IRGS, using a hardware-accelerated 2D ray tracer to refine material parameters based on the discrepancy between rendered reflections and observed images.

Why It Works

Differentiable path tracing works because it replaces "learning to render" with "learning the scene." In a traditional neural renderer, the network must learn both what the scene looks like and how light works. In a differentiable physics-based system, the "how light works" part is hard-coded as a differentiable simulator (the path tracer).

As Jijiang Li notes in DPCS, replacing black-box models with explicit light transport allows for much higher interpretability and requires significantly less training data. The optimizer doesn't have to figure out that light travels in straight lines or that mirrors reflect; it only has to find the material and geometry parameters that, when passed through the laws of physics, result in the image we see.

Papers in this version

Showing 60 of 97 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_differentiable_path_tracing_for_inverse__8a8c783e,
  title = {Differentiable Path Tracing for Inverse Rendering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/differentiable-path-tracing-for-inverse-rendering/rdr_8a8c783e660b4ebab8e35acb93c3c0cd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_differentiable_path_tracing_for_inverse__8a8c783e_v1,
  title = {Differentiable Path Tracing for Inverse Rendering},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/differentiable-path-tracing-for-inverse-rendering/rdr_8a8c783e660b4ebab8e35acb93c3c0cd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}