Hybrid Physics and Neural Rendering for Photorealistic Synthesis

This direction is split into other directions: hybrid neural-physical rendering. Its last version stays readable and citable here.

Version 1 of 1132 papers178 conceptsfrozen 2026-10-04

Computer graphics has long been a tug-of-war between two philosophies. On one side is Physically-Based Rendering (PBR), which uses deterministic optics—like the Beer-Lambert law for light absorption or Fresnel equations for reflections—to simulate how photons interact with matter. On the other side is Neural Rendering, which uses high-capacity models like Neural Radiance Fields (NeRF) or Generative Adversarial Networks (GANs) to learn appearance directly from data.

While PBR is controllable and physically consistent, it often looks "uncanny" because real-world geometry is rarely perfect. Conversely, neural methods produce stunning photorealism but often fail to maintain physical logic; shadows might not move correctly when a light source shifts, and energy conservation is frequently ignored.

A new research direction has emerged to resolve this tension: Hybrid Physics-Neural Rendering. This approach uses physical laws as a structural "scaffold" while delegating the "un-modelable" complexities of the real world to neural networks.

Physics as a Structural Scaffold

The core intuition behind this direction is that we shouldn't ask a neural network to learn what we already know from physics. For example, the way light intensity falls off over distance follows a strict inverse-square law. In Deep Near-Light Photometric Stereo for Spatially Varying Reflectances, Hiroaki Santo and Yasuyuki Matsushita offload these geometric light transport calculations to a differentiable physical model. This allows the neural component to focus entirely on the harder problem: learning complex material reflectances.

Similarly, LightSim, developed by Ava Pun and Jingkang Wang, builds "lighting-aware digital twins" for autonomous driving. It uses a physics engine to calculate geometric buffers and shadow maps. These maps act as a guide for a neural deferred renderer, ensuring that when a car moves, its shadow follows the laws of geometry even if the final pixels are synthesized by a network.

The Neural Residual

Even with perfect geometry, traditional shading models like Lambertian or Blinn-Phong are too rigid to capture effects like inter-reflections, soft shadows, or the subtle glow of skin. To solve this, researchers use a Neural Residual—a learned layer that "corrects" the output of a simplified physical model.

In Self-supervised Outdoor Scene Relighting, Abhimitra Meka and Ye Yu decompose an image into albedo and normals. They use these to create a base rendering and then apply a neural synthesis module to capture non-Lambertian effects like specularities. This ensures the image is grounded in physics while achieving the high-frequency detail of a photograph.

Thomas Nestmeyer applied a similar logic to human faces in Learning Physics-Guided Face Relighting Under Directional Light. By explicitly modeling the diffuse component and then predicting a "residual" for shadows and highlights, the system can relight "in-the-wild" portraits with a level of realism that purely analytical models cannot reach.

Modeling Complex Light Transport

Some phenomena, like Subsurface Scattering (SSS)—where light enters a translucent object, bounces around, and exits at a different point—are computationally ruinous to simulate with pure physics. Chenhao Li and Hajime Nagahara addressed this in their work on Inverse Rendering of Translucent Objects. They use a differentiable physical renderer for direct surface reflection but a neural network to approximate the internal scattering. This hybrid approach bypasses the need to solve the complex Radiative Transport Equation while maintaining the visual "waxy" look of translucent materials.

This philosophy extends to specialized domains like medical imaging. In PR-ENDO, Joanna Kaleta uses neural networks to handle the complex diffuse scattering of organic tissues in endoscopic videos, while relying on physical models for the sharp specular reflections caused by the camera's co-located light source.

Integrating Dynamic Simulations

The most recent frontier in this area is the integration of classical simulation engines (fluids, weather, combustion) directly into neural volumetric representations like NeRF. ClimateNeRF fuses NeRF with physical simulations to render controllable floods and blizzards. By integrating "simulatable" geometry with physics engines, it allows for the precise adjustment of physical parameters—like water levels or smog density—within a photorealistic 3D scene.

Even for chaotic phenomena like fire, researchers are finding that physics provides the necessary constraints. In Gaussians on Fire, Dominik Michels and Jakob Nazarenus suggest that incorporating combustion physics—the conversion of fuel into heat and light—into the optimization of 3D Gaussian Splatting is the key to capturing the high-frequency, emissive nature of flames.

By treating physics as the "logic" and neural networks as the "texture," this research direction provides a path toward digital twins that are not only indistinguishable from reality but also behave according to its rules.

Papers in this version

Showing 60 of 132 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_hybrid_physics_and_neural_rendering_for__a4efaeff,
  title = {Hybrid Physics and Neural Rendering for Photorealistic Synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hybrid-physics-and-neural-rendering-for-photorealistic-synthesis/rdr_a4efaeff75914a2d8261f92a29c03cc3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_hybrid_physics_and_neural_rendering_for__a4efaeff_v1,
  title = {Hybrid Physics and Neural Rendering for Photorealistic Synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hybrid-physics-and-neural-rendering-for-photorealistic-synthesis/rdr_a4efaeff75914a2d8261f92a29c03cc3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}