Abstract
Unsupervised learning with generative models has the potential of discovering rich representations of 3D scenes. While geometric deep learning has explored 3D-structure-aware representations of scene geometry, these models typically require explicit 3D supervision. Emerging neural scene representations can be trained only with posed 2D images, but existing methods ignore the three-dimensional structure of scenes. We propose Scene Representation Networks (SRNs), a continuous, 3D-structure-aware scene representation that encodes both geometry and appearance. SRNs represent scenes as continuous functions that map world coordinates to a feature representation of local scene properties. By formulating the image formation as a differentiable ray-marching algorithm, SRNs can be trained end-to-end from only 2D images and their camera poses, without access to depth or shape. This formulation naturally generalizes across scenes, learning powerful geometry and appearance priors in the process. We demonstrate the potential of SRNs by evaluating them for novel view synthesis, few-shot reconstruction, joint shape and appearance interpolation, and unsupervised discovery of a non-rigid face model.
Introduction
Teaching AI to understand the 3D structure of the world from 2D images is a massive challenge in computer vision. Prior approaches generally fall into two flawed categories: 2D neural renderers, which lack explicit 3D understanding and struggle with multi-view consistency, and discrete 3D representations (like voxel grids), which are limited by memory scaling and often require hard-to-obtain 3D ground-truth data. Scene Representation Networks (SRNs) address these gaps by proposing a continuous, memory-efficient 3D representation that strictly enforces multi-view geometry while being trainable entirely from standard 2D photographs.
Methods
The SRN architecture relies on several interconnected components that allow for end-to-end training from 2D images and camera poses.
- Implicit Continuous Representation: Instead of a discrete grid, a scene is represented as a continuous function (a Multi-Layer Perceptron, or MLP). This network maps a 3D world coordinate $(x, y, z)$ to a feature vector representing local properties like color and geometry.
- Differentiable Ray Marching (RM-LSTM): To render an image, the model shoots virtual camera rays into the 3D space. The authors introduce a learned ray-marching algorithm powered by a Long Short-Term Memory (LSTM) network that predicts optimal step lengths to find surface intersections efficiently.
- Per-Pixel Generator: Once a ray hits a surface, the extracted feature vector is passed to a pixel generator (an MLP acting as a 1x1 convolution). Because the color depends only on the 3D coordinate, the renders are mathematically guaranteed to be multi-view consistent.
- Hypernetworks for Generalization: To learn priors across object classes, an auto-decoder framework uses latent codes and Hypernetworks to generate weights for the scene representation MLP.
Datasets
The authors evaluated SRNs on several datasets to test generalization and reconstruction capabilities:
- ShapeNet v2: Used the "chairs" and "cars" classes for novel view synthesis and few-shot reconstruction.
- Shepard-Metzler Objects: Simple 3D block structures used to test limited-data learning.
- Basel Face Model: Used to test the model's ability to handle non-rigid facial expressions.
- DeepVoxels Dataset: Synthetic objects used for direct benchmarking against existing methods.
Key Results and Insights
A striking insight of SRNs is their ability to learn highly accurate 3D geometry (surface normals and depth) in a completely unsupervised manner. The network discovers 3D shape because it is the most logical way to explain the 2D images observed from different camera poses.
- State-of-the-Art View Synthesis: SRNs significantly outperformed baselines, achieving 26.23 dB PSNR on ShapeNet chairs compared to 22.72 dB for a deterministic GQN.
- Efficiency: Compared to DeepVoxels, SRNs achieved a 3 dB improvement in quality while using three orders of magnitude fewer parameters (~550k vs ~160M).
- Few-Shot Reconstruction: The model successfully reconstructed 3D objects from just 1 or 2 reference images by leveraging learned shape priors.
- Non-Rigid Animation: SRNs successfully learned to animate 3D faces, smoothly interpolating between expressions never explicitly seen paired with specific identities.
Limitations and Future Work
Current SRNs bake lighting and texture into a single representation, failing to model view-dependent effects like specular highlights. The learned ray marcher can occasionally fail near occluding surfaces, and the per-ray MLP evaluation is computationally expensive. Future directions include scaling to complex room-scale environments and incorporating probabilistic frameworks to handle ambiguous 2D inputs.
Summary updated