Novel View Synthesis via Depth-guided Skip Connections

Yuxin Hou, Arno Solin, Juho Kannala

Researchers from Aalto University have introduced a novel architectural approach to address the inherent trade-offs in Novel View Synthesis (NVS). Their method combines the structural stability of generative models with the high-fidelity detail found in image-based rendering.

The Challenge of Novel View Synthesis

Novel View Synthesis involves generating a scene from a new camera viewpoint using one or more source images. Historically, this task has been approached through two primary strategies, each with significant drawbacks:

  1. Image-Based Rendering (IBR): These methods warp pixels from the source image to the target view. While they preserve sharp textures, they often produce severe geometric distortions (the "melting" effect) when the underlying geometry is complex or occlusions occur.
  2. Pixel Generation: These methods use neural networks to synthesize the target image from a compressed latent code. While they maintain structural consistency, they typically produce blurry results because a single latent vector cannot capture every fine-grained texture detail.

The authors propose a hybrid solution: Depth-guided Skip Connections, designed to achieve the sharpness of IBR while maintaining the structural robustness of pixel generation.

Methodology: Depth-guided Skip Connections

Standard U-Net architectures utilize skip connections to pass high-resolution feature maps from the encoder to the decoder. However, in NVS, the input and output are spatially misaligned due to the change in viewpoint. To resolve this, the authors developed a pipeline that geometrically aligns these features before they reach the decoder.

The proposed architecture uses a predicted depth map to warp source features into the target coordinate frame.

The Pipeline:

  • Encoder: Extracts feature maps and a latent code from the source image.
  • Latent Transformation: The desired camera rotation and translation $[R|t]$ are applied directly to the latent code.
  • Depth Decoder: This transformed code predicts a depth map for the target view.
  • Warped Skip Connections: Instead of passing source features directly, the model uses the predicted target depth to geometrically warp source features into the target's coordinate frame via bilinear sampling.
  • Pixel Decoder: The decoder receives these aligned, high-resolution features to synthesize the final image, effectively filling in gaps where warping might be incomplete.

Training is conducted without ground-truth depth, utilizing a multi-scale reconstruction loss, a VGG perceptual loss for sharpness, and an edge-aware smoothness loss to ensure clean depth maps.

Experimental Results

The model was evaluated on two distinct datasets: ShapeNet (Chairs and Cars) for 3D structural complexity and KITTI Visual Odometry for real-world street scene textures.

Qualitative comparison on ShapeNet dataset showing the proposed method against various baselines.

Key Findings:

  • Feature Warping vs. Pixel Warping: By warping features within the network rather than raw pixels, the decoder acts as a learned refiner, correcting minor geometric distortions that would otherwise ruin an IBR output.
  • Quantitative Superiority: On ShapeNet Chairs, the method achieved an SSIM of 0.9256 and an L1 error of 0.0584, outperforming both pure pixel generation and IBR-heavy baselines.
  • Qualitative Fidelity: Visual results demonstrate that the method maintains sharp textures (such as fabric patterns) while keeping the overall geometry intact, avoiding the "ghostly" blur of generative models and the "broken" structures of IBR.

Conclusion and Future Work

By integrating geometric priors directly into the skip connections of a generative network, the authors have bridged the gap between structural accuracy and texture fidelity in NVS. While the method excels at general synthesis, thin structures remain a challenge due to the resolution limits of the predicted depth maps. Future research may focus on higher-resolution depth estimation or multi-view integration to better handle complex occlusions.

Summary updated

BibTeX
@inproceedings{hou2021novel,
  author = {Yuxin Hou and Arno Solin and Juho Kannala},
  title = {{Novel View Synthesis via Depth-guided Skip Connections}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00316},
}
Figure