Abstract
DiffusionRenderer is a unified neural framework that tackles both inverse and forward rendering by repurposing video diffusion models. It bypasses the need for explicit 3D geometry and precise path tracing, enabling high-quality relighting, material editing, and object insertion directly from a single video input. By leveraging the strong priors of large-scale video diffusion models, the framework "hallucinates" accurate light transport from imperfect 2D inputs while simultaneously improving the inverse rendering process.
Introduction
Physically-based rendering (PBR) remains the gold standard for photorealistic image generation. However, accurate simulation of light transport—including shadows and inter-reflections—requires a complete scene description: explicit 3D geometry, high-quality material properties (G-buffers), and precise lighting conditions. Extracting these parameters from real-world videos, known as inverse rendering, is a significant challenge. Traditional PBR engines or screen-space ray tracing (SSRT) often produce glaring artifacts when provided with the noisy, estimated 2D G-buffers typical of inverse rendering.
DiffusionRenderer addresses these challenges by utilizing video diffusion models to bridge the gap between imperfect scene estimations and high-quality rendering.
Methodology
The DiffusionRenderer framework consists of two primary components built upon the Stable Video Diffusion (SVD) architecture: the Neural Inverse Renderer and the Neural Forward Renderer.
Neural Inverse Renderer
This model estimates intrinsic scene attributes—such as normals, depth, albedo, roughness, and metallic properties—from an input RGB video. To ensure high quality, the model generates one attribute at a time, guided by an optimizable "domain embedding" that specifies the target G-buffer output. Using video data instead of single images allows the model to better untangle view-dependent properties like specular highlights by observing how light shifts across frames.
Neural Forward Renderer
Acting as a neural replacement for a traditional path tracer, this model takes estimated G-buffers and a target HDR environment map to generate a photorealistic video. To handle HDR lighting effectively, the authors implemented a custom environment map auto-encoder. This component extracts multi-resolution features from the HDR map and injects them into the diffusion UNet via cross-attention layers, providing precise control over light intensity and direction.
Training and Datasets
To overcome the "sim-to-real" gap, the authors employed a dual-training strategy. The models were initially trained on a synthetic dataset of 150,000 videos rendered from 36,500 Objaverse 3D assets. Subsequently, the inverse renderer was used to "auto-label" 150,000 real-world video segments from the DL3DV10k dataset. The forward renderer was then co-trained on both synthetic and auto-labeled real data, utilizing a Low-Rank Adaptation (LoRA) module to adapt to real-world aesthetics.
Quantitative evaluation was performed on two new datasets: SyntheticObjects (30 objects under varying light) and SyntheticScenes (40 complex scenes with inter-reflections).
Key Results and Insights
DiffusionRenderer demonstrates significant improvements over existing methods:
- Forward Rendering: On the
SyntheticScenesdataset, the model achieved a PSNR of 26.0, outperforming SSRT (24.8) and neural baselines like RGB↔X (18.5). Its performance remained robust even in complex scenes with significant inter-reflections. - Inverse Rendering: Leveraging temporal video data reduced the Root Mean Square Error (RMSE) for metallic properties by 41% and roughness by 20% compared to image-only baselines.
- Relighting: User studies showed a strong preference (up to 90%) for DiffusionRenderer's results over state-of-the-art methods like Neural Gaffer and DiLightNet.
A key insight is that neural rendering is inherently robust to noise. Unlike classic PBR, the diffusion-based forward renderer acts as a strong prior, synthesizing plausible shadows and reflections even from imperfect 2D inputs.
Conclusion
DiffusionRenderer represents a significant step toward practical, high-quality neural rendering from video. By integrating inverse and forward rendering within a unified diffusion-based framework, it achieves photorealistic results that were previously difficult to obtain from real-world captures. Future work will focus on improving inference speed through distillation and strictly enforcing content consistency during generative editing tasks.