Monocular Neural Image Based Rendering with Continuous View Control

Xu Chen, Jie Song, Otmar Hilliges

TL;DR

This paper introduces a novel neural network architecture for generating realistic novel views of 3D scenes from a single input image, allowing for precise, continuous control over the camera's 6-DOF viewpoint. By integrating a "transforming auto-encoder" with depth-guided appearance warping, the method learns to implicitly understand scene geometry without explicit 3D supervision, achieving impressive view synthesis quality and generalization.

What Problem Does This Paper Address?

Creating immersive 3D experiences often requires rendering novel views of objects or scenes. Traditional methods either rely on multiple input views, which is impractical for many real-world scenarios, or require storing and rendering full 3D information, which is computationally expensive and doesn't scale well. Existing learning-based approaches for monocular view synthesis often struggle with generalization to unseen viewpoints, leading to image quality degradation as the camera moves away from the training views. This paper tackles the challenge of generating geometrically accurate and high-quality novel views from a single image with fine-grained, continuous camera control, bridging the gap between 2D imagery and interactive 3D exploration.

Methods

The core of the proposed method is a novel geometry-aware neural architecture composed of four key components:

  1. 3D Transforming Auto-encoder (TAE): This component encodes the input image into a latent representation that is explicitly designed to be meaningful in 3D space. Crucially, it injects geometric transformations (3D rotations and translations) directly into the latent space. This forces the network to learn a representation that is equivariant to these transformations, implicitly capturing underlying geometry.
  2. Self-supervised Depth Map Prediction: The transformed latent code is then decoded to predict a depth map of the scene from the target viewpoint. This is a key step, as it provides geometric information without requiring explicit depth supervision.
  3. Depth Map Projection and Appearance Warping: The predicted depth map is used to compute dense correspondences between pixels in the source and target views via perspective projection. These correspondences are then used to warp the source image pixels into the target view. This depth-guided warping ensures that texture and local details are propagated accurately.
  4. Differentiable Bilinear Sampling: Since the computed pixel correspondences are often non-integer, differentiable bilinear sampling is used to ensure that the entire pipeline is end-to-end trainable via backpropagation.

The entire network is trained using a simple L1 reconstruction loss between the synthesized target view and the ground truth image, making it self-supervised.

Datasets

The paper evaluates its method on two datasets:

  • ShapeNet [3]: A collection of 3D synthetic objects from various categories (cars and chairs). For training, they used rendered views with specific azimuth and elevation ranges.
  • KITTI [15]: A standard dataset for autonomous driving, containing real-world city scenes. They used the odometry subset, which includes image sequences and global camera poses.

Key Insights and Ideas

  • Implicit Geometry Reasoning: By injecting geometric transformations into the latent space of an auto-encoder, the network is implicitly forced to learn about the underlying 3D structure of the scene, even without explicit 3D supervision.
  • Depth-Guided Warping for Accuracy: Using a predicted depth map to guide the warping process is more robust than directly predicting flow maps, as it enforces geometric constraints and leads to more accurate view synthesis, especially for non-training viewpoints.
  • Continuous Control from Discrete Training: The method effectively bridges the gap between discrete training data (specific view pairs) and continuous viewpoint control, overcoming the "snapping to training views" problem seen in prior work.
  • Generalization to Unseen Data: The learned representation and depth-guided warping enable the model to generalize well to entirely unseen images, including high-resolution internet images, opening up new application possibilities.

Key Results

  • Quantitative Performance: On ShapeNet, the proposed method achieved significantly lower L1 error and higher SSIM scores compared to state-of-the-art baselines, particularly for non-training views. For example, the full model achieved an average L1 error of 0.039 and SSIM of 0.949 on cars, outperforming others.
  • Qualitative Results: Visualizations demonstrate that the method produces sharp, geometrically accurate novel views with smooth transitions, closely matching ground truth. In contrast, other methods often produce distorted images or snap to fixed training views.
  • KITTI Evaluation: On the KITTI dataset, the method produced sharp and correct images for forward motion, and importantly, enabled accurate recovery of complex camera trajectories when used with a visual odometry system, indicating strong geometric consistency.
  • Depth and Flow Prediction: The method also achieved state-of-the-art accuracy in predicting depth maps and flow, which are crucial for view synthesis.

Limitations and Future Work

  • Limitations: While the method shows impressive generalization, it's noted that if large parts of an object are occluded in the source view, a purely learned prior is used, which might not always be perfect. The paper also mentions that SURF feature detection failed on blurry images from one baseline method, necessitating its removal from certain comparisons.
  • Future Work: The authors suggest that the model's ability to generalize to unseen images and its real-time inference speed (50ms per frame on a Titan X GPU) enable many appealing applications, such as bringing downloaded 2D product images to life for interactive browsing or creating free-viewpoint videos from single images for AR/VR content. The paper's core contribution of a robust, self-supervised monocular view synthesis method with fine-grained control lays the groundwork for these future applications.

Summary updated

Abstract

We present an approach that learns to synthesize high-quality, novel views of 3D objects or scenes, while providing fine-grained and precise control over the 6-DOF viewpoint. The approach is self-supervised and only requires 2D images and associated view transforms for training. Our main contribution is a network architecture that leverages a transforming auto-encoder in combination with a depth-guided warping procedure to predict geometrically accurate unseen views. Leveraging geometric constraints renders direct supervision via depth or flow maps unnecessary. If large parts of the object are occluded in the source view, a purely learning based prior is used to predict the values for dis-occluded pixels. Our network furthermore predicts a per-pixel mask, used to fuse depth-guided and pixel-based predictions. The resulting images reflect the desired 6-DOF transformation and details are preserved. We thoroughly evaluate our architecture on synthetic and real scenes and under fine-grained and fixed-view settings. Finally, we demonstrate that the approach generalizes to entirely unseen images such as product images downloaded from the internet.
BibTeX
@inproceedings{chen2019monocular,
  author = {Xu Chen and Jie Song and Otmar Hilliges},
  title = {{Monocular Neural Image Based Rendering with Continuous View Control}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00419},
}

Loading related content...

Figure