Computer graphics has traditionally relied on Physics-Based Rendering (PBR) to synthesize images. By mathematically simulating how light rays bounce off surfaces, interact with materials, and pass through lenses, PBR guarantees physical consistency, realistic shadows, and perfect controllability. However, modeling every microscopic scratch, hair strand, or atmospheric particle is computationally exhausting and requires meticulously designed 3D assets.
Conversely, modern neural rendering techniques like Neural Radiance Fields (NeRF) and 3D Gaussian Splatting learn to synthesize scenes directly from images. While they produce stunning photorealism, they operate largely as "black boxes." Because they learn statistical correlations rather than causal physics, they often bake lighting directly into geometry. Try to move a light source or view the scene from an extreme angle, and the illusion shatters.
To solve this, a powerful paradigm has emerged: hybrid neural-physical rendering. By embedding explicit physical laws directly into neural architectures, researchers are building systems that achieve the photorealistic flexibility of deep learning alongside the strict controllability and generalization of classical physics.
Inductive Biases via Multiplicative Decomposition
In standard deep learning pipelines, features are typically combined using additive operations or channel concatenation. While flexible, this design ignores the fundamental nature of light transport. In the real world, the color of a pixel is primarily a multiplicative interaction: the incoming light intensity multiplied by the surface's reflectance (albedo).
To capture this intuition, Guoxian Song et al. in Half-body Portrait Relighting with Overcomplete Lighting Representation demonstrate that combining lighting codes and subject features via element-wise multiplication across multiple scales drastically improves rendering quality. This simple architectural shift mimics physical image formation, resulting in significantly sharper shadows and eliminating the "ghosting" artifacts common to concatenative networks.
Standard Neural Layer: [Features] + [Light Code] --> Additive/Linear Blending (Blurry Shadows)
Physics-Informed Layer: [Features] × [Light Code] --> Multiplicative Interaction (Sharp Shadows)
Forcing a neural network to adhere to these optical laws acts as a powerful geometric regularizer. For instance, Anurag Ranjan in FaceLit: Neural 3D Relightable Faces embeds a physics-based reflectance model directly into a neural volume rendering pipeline. Instead of allowing the network to output arbitrary color values, the generator is forced to output explicit physical properties: albedo, specular reflectance, and surface normals. Because the final shading is computed through a deterministic optical equation, the underlying network is constrained to learn highly accurate, 3D-consistent geometry that naturally generalizes to unseen lighting directions.
The Hybrid Pipeline: Scaffolding and Residuals
We do not need neural networks to learn things we can already calculate perfectly with a calculator. Computing how light intensity drops off over a distance follows a strict inverse-square law; delegating this calculation to a neural network wastes capacity and invites errors.
Hybrid rendering architectures exploit this by using classical physics as a structural scaffold, leaving the neural network to focus entirely on high-dimensional pattern matching. As shown by Hiroaki Santo et al. in Deep Near-Light Photometric Stereo for Spatially Varying Reflectances, offloading geometric light transport formulas to a differentiable physical model allows the neural component to specialize in decoding complex, spatially varying material reflectances.
A common manifestation of this strategy is the physics-plus-residual architecture:
- The Physical Base: A deterministic graphics equation calculates a baseline image (e.g., direct diffuse lighting and basic geometric shadows).
- The Neural Residual: A neural network predicts a correction map to account for complex, mathematically intractable phenomena (e.g., subsurface scattering, inter-reflections, and hardware-specific sensor noise).
Thomas Nestmeyer demonstrates the elegance of this approach in Learning Physics-Guided Face Relighting Under Directional Light. The first stage uses a physics-based image formation model to decompose a portrait into intrinsic components for initial rendering. The second stage then predicts a neural residual to correct for non-diffuse elements like specular highlights and complex cast shadows.
Similarly, in large-scale urban simulations, Ava Pun et al. introduce a hybrid approach in Neural Lighting Simulation for Urban Scenes. Here, a traditional physics engine computes geometric buffers and shadow maps to provide a rigid structural scaffold. A neural deferred renderer then ingests these physically consistent maps to synthesize photorealistic textures and complex global illumination, bypassing the spatial inconsistencies that plague purely generative models.
[Scene Geometry & Light]
│
├──► Classical Physics Engine ──► Baseline Shading & Shadows ┐
│ ▼
└──► Neural Network ──► High-Frequency Residual ──► (+) ──► Final Photorealistic Image
This division of labor is especially vital for organic or intricate structures like hair and tissue, where full physical simulation is impossible in real time. Yang Zheng et al. in GroomLight: Hybrid Inverse Rendering for Relightable Human Hair Appearance Modeling pair an analytical Bidirectional Scattering Distribution Function (BSDF) with a neural residual layer. The physical model ensures that the hair generalizes across novel lighting conditions, while the neural layer captures the complex multiple-scattering effects that analytical formulas miss.
In medical imaging, Joanna Kaleta et al. apply this concept to endoscopy in PR-ENDO: Physically Based Relightable Gaussian Splatting for Endoscopy, mapping specular reflections with predictable physical laws while using a multi-layer perceptron (MLP) to handle the complex, organic scattering of light through living tissue.
Parametric Fields vs. Raw Color Prediction
When designing a neural scene representation, a critical question arises: what exactly should the network output? Early neural renderers predicted raw RGB color values directly from a given viewpoint. However, this approach struggles with view extrapolation; if a camera moves to an angle where a glossy highlight should appear, a raw RGB network has no physical basis for predicting it.
To solve this, hybrid methods use parametric neural architectures. Instead of predicting color, the network outputs the coefficients of analytical reflectance models, such as Spherical Gaussians or microfacet BRDFs.
Daniel Cremers et al. provide a thorough head-to-head analysis of this choice in On Neural BRDFs: A Thorough Comparison of State-of-the-Art Approaches. They show that while purely neural models excel at memorizing complex reflections within a training set, parametric neural architectures are vastly more robust to real-world noise and preserve physical reciprocity.
By understanding the underlying physics of reflection, these models can accurately extrapolate how highlights and shadows shift when viewed from angles far outside the original training data, as demonstrated by Kai Zhang et al. in PhySG: Inverse Rendering With Spherical Gaussians for Physics-Based Material Editing and Relighting.
Modeling Participating Media and Environmental Physics
The synergy between neural representations and physical models extends beyond solid surfaces to fluid dynamics and atmospheric optics. In scenes containing fog, smoke, or fire, purely data-driven models struggle because the geometry itself is volumetric, translucent, and constantly shifting.
To reconstruct scenes through adverse weather, Andrea Ramazzina et al. develop ScatterNeRF: Seeing Through Fog with Physically-Based Inverse Neural Rendering. By integrating the classical Koschmieder atmospheric scattering model directly into a neural radiance field, they enable the system to simultaneously estimate scene geometry and medium density. Because the physics of haze is explicitly modeled, the network can cleanly separate the clear 3D scene from the scattering atmosphere without requiring massive synthetic training sets.
Hazy Video Input ──► [ NeRF + Koschmieder Scattering Model ] ──► Clear 3D Scene Geometry
└──► Isolated Fog Density Map
For dynamic environmental synthesis, the authors of ClimateNeRF: Extreme Weather Synthesis in Neural Radiance Field fuse learned 3D scenes with deterministic physics equations like the Beer-Lambert law for participating media and Fresnel equations for water surfaces. This allows users to simulate extreme weather events—such as floods or heavy smog—with perfect physical accuracy and user controllability over parameters like fluid levels or atmospheric density.
Taking this a step further, Qianfan Shen et al. introduce FieryGS: In-the-Wild Fire Synthesis with Physics-Integrated Gaussian Splatting, coupling incompressible flow models with neural radiance fields. By anchoring the neural rendering to fluid dynamics equations, they achieve highly realistic, user-controllable combustion and smoke effects that remain bound by the laws of thermodynamics.
Why the Hybrid Approach Works
Hybrid neural-physical rendering succeeds because it perfectly balances the strengths and weaknesses of its constituent parts:
- Physics provides the global constraints: It handles low-frequency correctness, ensures energy conservation, dictates how shadows fall, and enables robust extrapolation to unseen views and lighting conditions.
- Neural networks provide the high-frequency details: They absorb real-world noise, learn complex material quirks, and efficiently approximate multi-bounce light transport that would take hours to compute via traditional ray tracing.
By treating physical laws not as obstacles to replace, but as structural scaffolds to build upon, hybrid rendering bridges the gap between the mathematical precision of traditional computer graphics and the expressive power of deep learning.
Key Foundational Work
Learning Physics-Guided Face Relighting Under Directional Light (2020) — contributed 2 concept elements to this direction.
Seeing the World in a Bag of Chips (2020) — contributed 2 concept elements to this direction.
DeLTra: Deep Light Transport for Projector-Camera Systems (2020) — contributed 2 concept elements to this direction.