Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations

Vincent Sitzmann, Michael Zollhöfer, Gordon Wetzstein

Overview

Scene Representation Networks (SRNs) represent 3D scenes as continuous neural functions that map 3D coordinates to local features. By combining this representation with a novel differentiable ray-marching algorithm, SRNs can learn complex 3D geometry and appearance from 2D images alone, significantly outperforming previous "black-box" neural renderers.

The Challenge of 3D Understanding

In computer vision and graphics, teaching neural networks to understand the 3D world is a fundamental goal. However, most existing approaches fall into two flawed camps:

  1. Discrete Representations: Methods using voxel grids, point clouds, or meshes are limited by resolution and memory. They often require explicit 3D supervision (like 3D scans), which is expensive and difficult to obtain.
  2. Black-Box Neural Renderers: Earlier "neural scene" models (like the Generative Query Network) treat 3D structure as an implicit latent variable. Because they don't explicitly enforce the laws of perspective and geometry, they often struggle with multi-view consistency—the same object might look different when viewed from a new angle—and fail to generalize when training data is sparse.

The authors of this paper ask: Can we create a representation that is continuous, memory-efficient, and mathematically aware of 3D structure, while only requiring 2D images for training?

Scene Representation Networks (SRNs)

The core of the paper is the SRN, which consists of three main components:

  • The Scene Function ($\Phi$): Instead of a grid of voxels, the scene is represented by a Multi-Layer Perceptron (MLP). Given a 3D coordinate $(x, y, z)$, it outputs a feature vector representing the properties of the scene at that exact point.
  • Differentiable Ray-Marching: To "render" an image, the model must find where a camera ray hits the scene surface. The authors use a specialized LSTM (Long Short-Term Memory) network to learn how to step along a ray. This is a "learned" version of sphere tracing; the LSTM predicts the step length based on the features at the current 3D point.
  • Pixel Generator: Once the ray-marcher finds the surface intersection, a final per-pixel MLP maps the local feature vector to an RGB color. Crucially, this happens independently for every pixel without using 2D convolutions, ensuring that the color of a 3D point remains consistent regardless of the camera's viewpoint.
Architecture of the Scene Representation Network

To generalize across different objects (e.g., learning what "chairs" look like in general), the authors use a Hypernetwork. This meta-network takes a latent code representing a specific object and generates the weights for that object's specific Scene Function.

Key Insights

  • Geometry as an Emergent Property: SRNs are never shown a 3D model or a depth map during training. The 3D geometry "emerges" purely because it is the most efficient way for the network to explain the 2D images it sees from different camera poses.
  • Continuous is Better than Discrete: Because the scene is a function rather than a grid, it has theoretically infinite resolution. You can zoom in on a surface, and the model won't become "pixelated" or "blocky" like a voxel grid.
  • The Power of 1x1 Convolutions: By avoiding standard 2D convolutions in the renderer, the authors guarantee multi-view consistency. In a standard CNN, a pixel's value depends on its neighbors; in SRNs, a pixel's value depends only on the 3D point it represents.
  • Interpretability: Unlike black-box models, SRNs are interpretable. You can extract depth maps and normal maps (vectors showing surface direction) directly from the ray-marching process, allowing you to "see" what the network thinks the 3D shape looks like.
Extracted normal maps and rendered views of a non-rigid face model

Performance and Results

The authors evaluated SRNs on several benchmarks, including Shepard-Metzler, ShapeNet v2 (chairs and cars), DeepVoxels Objects, and the Basel Face Model.

  • Superior Novel View Synthesis: On ShapeNet "chairs," SRNs achieved a PSNR (image quality metric) of 26.23 dB, significantly higher than the 22.72 dB achieved by the deterministic version of the Generative Query Network (dGQN).
  • Few-Shot Reconstruction: SRNs can reconstruct a plausible 3D object from just one or two 2D images by leveraging the "priors" they learned during training.
  • Latent Interpolation: The model allows for smooth "morphing" between objects. You can take the latent code for a sedan and a truck and smoothly transition between them, with the 3D geometry updating consistently throughout.
  • Non-Rigid Deformation: As demonstrated with the Basel Face Model, SRNs can successfully model and render non-rigid deformations like facial expressions.

Limitations and Future Work

While groundbreaking, SRNs have limitations that pave the way for future research:

  • View-Dependency: The current model assumes surfaces look the same from all angles. It cannot yet handle shiny reflections (specularity) or transparency.
  • Entanglement: Shape and appearance are currently baked into the same feature vector; future work could separate these components.
  • Complexity: While SRNs can represent a single room-scale scene, scaling them to represent massive, cluttered, or highly complex environments remains an open challenge.
  • Probabilistic Modeling: The current version is non-probabilistic, meaning it produces a single "best guess" for a 3D shape rather than a range of possible shapes when given ambiguous 1-shot data.

Summary updated

Abstract

Unsupervised learning with generative models has the potential of discovering rich representations of 3D scenes. While geometric deep learning has explored 3D-structure-aware representations of scene geometry, these models typically require explicit 3D supervision. Emerging neural scene representations can be trained only with posed 2D images, but existing methods ignore the three-dimensional structure of scenes. We propose Scene Representation Networks (SRNs), a continuous, 3D-structure-aware scene representation that encodes both geometry and appearance. SRNs represent scenes as continuous functions that map world coordinates to a feature representation of local scene properties. By formulating the image formation as a differentiable ray-marching algorithm, SRNs can be trained end-to-end from only 2D images and their camera poses, without access to depth or shape. This formulation naturally generalizes across scenes, learning powerful geometry and appearance priors in the process. We demonstrate the potential of SRNs by evaluating them for novel view synthesis, few-shot reconstruction, joint shape and appearance interpolation, and unsupervised discovery of a non-rigid face model.
BibTeX
@inproceedings{sitzmann2019scene,
  author = {Vincent Sitzmann and Michael Zollhöfer and Gordon Wetzstein},
  title = {{Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=rQDf52w7ih1m},
}
Figure