Abstract
We present an approach that learns to synthesize high-quality, novel views of 3D objects or scenes, while providing fine-grained and precise control over the 6-DOF viewpoint. The approach is self-supervised and only requires 2D images and associated view transforms for training. Our main contribution is a network architecture that leverages a transforming auto-encoder in combination with a depth-guided warping procedure to predict geometrically accurate unseen views. Leveraging geometric constraints renders direct supervision via depth or flow maps unnecessary. If large parts of the object are occluded in the source view, a purely learning based prior is used to predict the values for dis-occluded pixels. Our network furthermore predicts a per-pixel mask, used to fuse depth-guided and pixel-based predictions. The resulting images reflect the desired 6-DOF transformation and details are preserved. We thoroughly evaluate our architecture on synthetic and real scenes and under fine-grained and fixed-view settings. Finally, we demonstrate that the approach generalizes to entirely unseen images such as product images downloaded from the internet.
Introduction: The Challenge of Monocular NVS
Novel View Synthesis (NVS) from a single 2D image is a fundamentally "ill-posed" problem, as infinite 3D shapes could potentially result in the same 2D projection. Existing deep learning models often struggle with fine-grained camera control, frequently "snapping" back to discrete training angles or producing blurry results when moving between known viewpoints. To achieve truly interactive, 6-degree-of-freedom (6-DoF) control, a model must implicitly understand underlying 3D geometry rather than relying on simple 2D image-to-image translation.
Methodology: Geometry-Aware Architecture
The proposed pipeline forces the network to reason about 3D space through a geometry-aware architecture consisting of four main stages:
- 3D Transforming Autoencoder (TAE): The encoder maps the source image into a latent 3D representation that is explicitly rotated and translated in Euclidean space based on desired camera movement.
- Self-Supervised Depth Prediction: This transformed latent code is decoded into a depth map of the target view.
- Depth-Guided Warping: Using the predicted depth and perspective projection math, the system calculates exactly where each pixel from the source image should move to appear in the target view.
- Differentiable Rendering: The final image is created via bilinear sampling. The entire model is trained end-to-end using only pairs of images, requiring no ground-truth 3D models or depth maps.
By predicting how to move pixels (warping) rather than generating them from scratch, the model preserves sharp textures and fine details.
Evaluation and Key Insights
The model was evaluated on ShapeNet (synthetic objects like cars and chairs) and KITTI (real-world street scenes). Several key insights emerged from this evaluation:
- Geometry as a Constraint: By injecting 3D math directly into the network's bottleneck, the latent space is forced to behave like real objects when rotated.
- Eliminating "Snapping": Explicit geometric transformations prevent the model from overfitting to discrete training angles, ensuring smooth transitions.
- Texture Invariance: Because the model warps the source image, the learned geometry is independent of color or texture, allowing it to generalize to unseen high-resolution images.
Results and Performance
- Superior Smoothness: On ShapeNet, the model produced spin patterns that closely matched ground truth, significantly outperforming baselines that struggle to move away from fixed training views.
- Quantitative Accuracy: The model achieved higher Structural Similarity (SSIM) and lower Mean Absolute Error (L1) than state-of-the-art methods, reducing L1 error for cars by approximately 30%.
- Trajectory Recovery: Visual odometry algorithms were able to accurately recover intended camera paths from synthesized KITTI videos, proving the views were geometrically consistent.
- Real-time Performance: Inference takes only 50ms per frame on a Titan X GPU, enabling interactive applications.
Limitations and Future Work
While effective, the model faces challenges with occlusions—when a camera move reveals parts of an object hidden in the original photo. These "holes" are currently filled by the warping process but can sometimes appear as stretched artifacts. Additionally, extremely cluttered or dynamic scenes remain difficult for monocular systems. Future work may integrate more sophisticated inpainting techniques to better handle dis-occluded regions for full 360-degree reconstructions.
Summary updated