Deferred neural rendering: image synthesis using neural textures

Justus Thies, Michael Zollhöfer, Matthias Nießner

Abstract

Deferred Neural Rendering addresses the limitations of traditional graphics pipelines and 2D generative models by introducing a hybrid synthesis approach. By storing learnable, high-dimensional feature maps—termed "Neural Textures"—on 3D proxy meshes, the method leverages a convolutional neural network to perform "deferred" rendering. This pipeline ensures 3D consistency while producing photo-realistic outputs from imperfect geometry. The approach is demonstrated across tasks including novel view synthesis, scene editing, and real-time facial animation.

Introduction

The traditional computer graphics pipeline is highly effective but requires meticulous modeling of geometry, materials, and lighting to achieve photo-realism. Conversely, 2D generative models like GANs produce high-quality images but often lack 3D structural consistency, leading to "swimming" artifacts during camera movement. Deferred Neural Rendering bridges this gap by using a rough 3D proxy to maintain spatial structure while employing a neural network to handle complex appearance modeling.

Methods

The core of the proposed system consists of two primary components: Neural Textures and a Deferred Neural Renderer.

Neural Textures

Unlike standard RGB textures, a Neural Texture stores a high-dimensional feature vector (typically 16 channels) at each texel. These features are learned during training to encode local appearance and geometric nuances that simple color maps cannot capture. As the network learns, the initial channels often represent a low-frequency approximation of the object's appearance.

Visualization of a neural texture feature map on a 3D proxy globe.

The Rendering Pipeline

The pipeline follows a "deferred" approach:

  1. Rasterization: A standard graphics pipeline identifies visible portions of the neural texture based on the camera view.
  2. Sampling: Differentiable sampling (using a Mipmap-like hierarchy) generates a screen-space "feature image."
  3. Neural Rendering: A U-Net-based convolutional neural network translates this feature image into a final RGB output.

The entire process is end-to-end differentiable, allowing the neural textures and the renderer to be optimized simultaneously against ground-truth imagery.

Key Results and Applications

The method demonstrates significant improvements in both quality and temporal stability compared to state-of-the-art image-to-image translation techniques.

Overview of Deferred Neural Rendering applications: Novel View Synthesis, Scene Editing, and Animation Synthesis.

Novel View Synthesis and Scene Editing

By anchoring features to a 3D surface, the system maintains perfect temporal consistency. This allows for seamless scene editing; objects can be added or removed by simply modifying the underlying proxy mesh. Quantitative tests show lower Mean Squared Error (MSE) than traditional Image-Based Rendering (IBR) methods.

Real-Time Animation Synthesis

The system is highly efficient, with the neural rendering stage taking approximately 4ms on an NVIDIA 1080Ti. This enables real-time facial reenactment, where the identity of a subject is preserved while their expressions are driven by a different source.

Conclusion

Deferred Neural Rendering provides a robust framework for photo-realistic synthesis that respects 3D constraints. While currently limited by scene-specific training and static lighting assumptions, it represents a significant step toward integrating learnable components into the graphics pipeline. Future work may explore generalized models and volumetric representations.

Summary updated

BibTeX
@article{thies2019deferred,
  author = {Justus Thies and Michael Zollhöfer and Matthias Nießner},
  title = {{Deferred neural rendering: image synthesis using neural textures}},
  journal = {ACM Trans. Graphics},
  year = {2019},
  doi = {10.1145/3306346.3323035},
}
Figure