Abstract
Generating smooth, continuous novel viewpoints from a single static image remains a significant challenge in computer vision. We present a geometry-aware neural rendering pipeline that achieves fine-grained camera control without the artifacts typical of previous methods. By integrating a 3D transforming autoencoder with depth-guided warping, our system produces mathematically consistent view synthesis for both isolated objects and complex real-world scenes.
Introduction
Monocular view synthesis—the task of generating new perspectives from a single 2D image—is inherently ill-posed due to the lack of explicit depth information. Traditional deep learning approaches often treat this as a black-box image-to-image translation problem, leading to two primary failure modes: significant blurriness in generated textures and a "snapping" effect where the model fails to interpolate smoothly between discrete training viewpoints.
Methodology: Geometry as a Constraint
To address these issues, we propose a pipeline that explicitly reasons about 3D geometry rather than treating view synthesis as a simple pixel-generation task.
3D Transforming Autoencoder (TAE)
Our architecture utilizes a 3D Transforming Autoencoder. Unlike standard autoencoders that compress images into unstructured latent vectors, the TAE maps the input image into a 3D latent space. This allows for the application of explicit Euclidean rotations and translations directly to the latent representation. By forcing the network to operate within a mathematically consistent 3D framework, we ensure that the synthesized views follow the laws of perspective projection.
Depth-Guided Appearance Mapping
To maintain high-frequency details and avoid the "soupy" appearance of directly generated pixels, we employ a warping-based approach. The model predicts a depth map for the target viewpoint. Using this depth map and standard camera intrinsics, we calculate the backward flow—mapping each pixel in the target frame back to its original position in the source image. Bilinear sampling is then used to "warp" the original pixels into the new frame, preserving the sharp textures of the source photo.
Experimental Results
The model was evaluated on two distinct benchmarks: ShapeNet (synthetic cars and chairs) and KITTI (real-world street scenes).
Continuous View Control
A key breakthrough is the ability to generate views at 1° increments, even when trained on much coarser 20° steps. Previous state-of-the-art methods, such as those by Zhou et al. and Sun et al., often "snap" to the nearest training angle or lose structural integrity during interpolation. Our method maintains geometric consistency throughout the entire rotation or translation path.
Quantitative and Qualitative Performance
On the ShapeNet "car" dataset, our method reduced Mean Absolute Error ($L_1$) by 30-50% compared to existing baselines and achieved significantly higher Structural Similarity (SSIM) scores. In real-world KITTI sequences, the generated "fly-through" videos were sufficiently consistent that a standard Visual Odometry algorithm could successfully track the camera's path through the synthesized imagery. Furthermore, the inference process is efficient, taking approximately 50ms per frame on a Titan X GPU.
Discussion and Limitations
By injecting 3D mathematical constraints into the network architecture, we prevent the model from relying on memorization. Furthermore, the explicit transformation of the latent space makes it interpretable; for instance, interpolating between the latent codes of different vehicle types results in a smooth transition of shape while maintaining correct orientation.
However, challenges remain. Like all warping-based methods, our model struggles with occlusions—areas visible in the target view but hidden in the source. Currently, the system does not explicitly "paint in" these missing regions. Additionally, the model is limited to a single source image and requires known camera intrinsics, which may limit its application to uncalibrated "in-the-wild" photography.
Summary updated