Abstract
Is a geometric model required to synthesize novel views from a single image? Being bound to local convolutions, CNNs need explicit 3D biases to model geometric transformations. In contrast, we demonstrate that a transformer-based model can synthesize entirely novel views without any hand-engineered 3D biases. This is achieved by (i) a global attention mechanism for implicitly learning long-range 3D correspondences between source and target views, and (ii) a probabilistic formulation necessary to capture the ambiguity inherent in predicting novel views from a single image, thereby overcoming the limitations of previous approaches that are restricted to relatively small viewpoint changes. We evaluate various ways to integrate 3D priors into a transformer architecture. However, our experiments show that no such geometric priors are required and that the transformer is capable of implicitly learning 3D relationships between images. Furthermore, this approach outperforms the state of the art in terms of visual quality while covering the full distribution of possible realizations. Code is available at https://git.io/JOnwn
Introduction
Novel View Synthesis (NVS) involves generating an image of a scene from a new camera perspective based on one or more source images. Traditionally, this task has been framed as a geometric problem. Convolutional Neural Networks (CNNs), due to their local receptive fields, often require explicit 3D inductive biases—such as depth-based warping or voxel grids—to handle geometric transformations.
However, these explicit methods struggle with significant viewpoint changes where large portions of the scene are occluded in the source view. Deterministic models typically produce blurry results in these regions as they attempt to predict an "average" of all possible outcomes. This paper investigates whether a model can learn to reason about 3D space and handle inherent uncertainty without hand-engineered geometric priors.
Methodology
The proposed approach treats NVS as a probabilistic sequence-to-sequence prediction task, implemented in two stages:
- VQGAN (Vector Quantized GAN): To manage the high dimensionality of image space, a VQGAN compresses images into a grid of discrete tokens. This allows the transformer to operate in a compressed latent space while maintaining the ability to reconstruct high-fidelity images.
- Autoregressive Transformer: A GPT-2 style transformer models the distribution of target image tokens. It is conditioned on the source image tokens and the camera transformation $T$ (comprising the intrinsic matrix $K$, rotation $R$, and translation $t$).
By predicting a probability distribution over tokens, the model can sample multiple plausible realizations of unseen regions, effectively "imagining" the scene beyond the source view's boundaries.
Key Insights: Attention as a 3D Proxy
A core finding of this work is that the global attention mechanism in transformers serves as an effective proxy for 3D geometry. It allows the model to learn long-range correspondences between source and target views without explicit warping layers.
To verify if the model truly "understands" geometry, the authors performed linear probing on the transformer's internal features. They discovered that accurate depth maps could be extracted using a simple linear layer, indicating that the transformer unsupervisedly learned to represent depth as a byproduct of solving the view synthesis task.
Evaluation and Results
The model was evaluated on the RealEstate10K and ACID (Aerial Coastline Imagery Dataset) datasets. A specific test split was created to evaluate challenging camera motions, such as large forward/backward and sideways movements.
The transformer-based approach significantly outperformed state-of-the-art methods like SynSin and 3DPhoto. On RealEstate10K, the implicit "nodepth" variant achieved an FID of 48.59, a substantial improvement over SynSin's 113.88. The probabilistic formulation ensures sharp, semantically consistent textures even in regions completely invisible in the source frame, avoiding the blurriness common in deterministic CNN-based approaches.
Conclusion
This research demonstrates that explicit 3D priors are not a prerequisite for high-quality novel view synthesis. By leveraging the global attention of transformers and a probabilistic framework, the model implicitly learns 3D relationships and handles the ambiguity of large viewpoint changes. While limitations such as VQGAN compression artifacts and temporal consistency in video generation remain, this work opens new avenues for geometry-free 3D scene understanding and synthesis.
Summary updated