View Independent Generative Adversarial Network for Novel View Synthesis

Xiaogang Xu, Ying-Cong Chen, Jiaya JiaICCV 2019 · OpenReview

Abstract

Synthesizing novel views from a 2D image requires inferring 3D structure and projecting it back to 2D from a new viewpoint. In this paper, we propose an encoder-decoder based generative adversarial network, VI-GAN, to tackle this problem. Our method allows the network, after seeing many images of objects belonging to the same category in different views, to obtain essential knowledge of the intrinsic properties of the objects. To this end, an encoder is designed to extract a view-independent feature that characterizes intrinsic properties of the input image, including 3D structure, color, and texture. We also make the decoder hallucinate the image of a novel view based on the extracted feature and an arbitrary user-specific camera pose. Extensive experiments demonstrate that our model can synthesize high-quality images in different views with continuous camera poses and is general for various applications.

Introduction

Novel view synthesis—generating an image of an object from a new perspective given a single source image—is a classic challenge in computer vision. Traditionally, researchers have approached this through geometry-based methods or learning-based methods. Geometry-based methods attempt to explicitly reconstruct 3D models, which is often ill-posed from a single 2D image and leads to distortions. Learning-based methods use neural networks to map views directly but often struggle with structural integrity and generalization to unseen viewpoints. VI-GAN bridges this gap by learning 3D structure implicitly, understanding an object's 3D essence without requiring explicit 3D supervision like voxels or point clouds during training.

Methodology: The VI-GAN Architecture

The core of the proposed system is the View Independent Generative Adversarial Network (VI-GAN). Its architecture follows an encoder-decoder structure designed to disentangle identity from pose.

The VI-GAN architecture overview, highlighting the encoder-decoder structure, dual discriminators, and the extraction of view-independent features.

Key Components

  • The Encoder ($E$): Extracts a "view-independent feature" representing the object's shape, color, and texture, independent of the camera's position.
  • The Decoder ($D$): Acts as a neural renderer, taking the view-independent feature and a target camera pose to synthesize the new image. It utilizes Adaptive Instance Normalization (AdaIN) to inject target camera poses, which significantly improves performance over standard concatenation.
  • Dual Discriminators: The system employs two discriminators: one for image realism and a Pose Prediction Discriminator to ensure the generated image matches the requested camera angle.
  • Loss Functions: Training involves a combination of losses, including a View-Independent Loss to ensure feature consistency across views and a Cycle Consistency Loss to maintain structural integrity during transformations.

Experiments and Results

VI-GAN was evaluated on three major benchmarks: ShapeNet (chairs, sofas, and benches), Multi-PIE (human faces at discrete angles), and 300W-LP (faces with continuous camera poses).

Performance on Complex Objects

On the ShapeNet dataset, VI-GAN achieved the lowest $L_1$ error and highest Structural Similarity Index (SSIM) compared to previous methods like MV3D and Appearance-Flow. Qualitatively, the model excels at "hallucinating" hidden parts of an object, such as the back of a chair, without the blurry artifacts common in earlier learning-based approaches.

Qualitative results on the ShapeNet dataset showing synthesized views of various chair models across different poses.

Facial Synthesis and Implicit 3D Knowledge

For facial synthesis on the Multi-PIE dataset, VI-GAN achieved a significantly lower FID score (6.51) compared to CR-GAN (8.76), indicating superior image quality. A key insight from these experiments is that the "view-independent feature" implicitly learns 3D geometry; the authors demonstrated that a simple linear model could predict 3D facial landmarks from these features with high accuracy.

Conclusion

VI-GAN demonstrates that forcing a latent space to be view-independent allows a model to effectively separate identity from pose. This disentanglement enables high-quality novel view synthesis with continuous camera movement. While the model currently requires category-specific training and can struggle with extreme occlusions, its ability to implicitly learn 3D structures suggests it could serve as a powerful pre-training step for broader 3D reconstruction tasks.

Summary updated

Abstract

Synthesizing novel views from a 2D image requires to infer 3D structure and project it back to 2D from a new viewpoint. In this paper, we propose an encoder-decoder based generative adversarial network VI-GAN to tackle this problem. Our method is to let the network, after seeing many images of objects belonging to the same category in different views, obtain essential knowledge of intrinsic properties of the objects. To this end, an encoder is designed to extract view-independent feature that characterizes intrinsic properties of the input image, which includes 3D structure, color, texture etc. We also make the decoder hallucinate the image of a novel view based on the extracted feature and an arbitrary user-specific camera pose. Extensive experiments demonstrate that our model can synthesize high-quality images in different views with continuous camera poses, and is general for various applications.
BibTeX
@inproceedings{xu2019view,
  author = {Xiaogang Xu and Ying-Cong Chen and Jiaya Jia},
  title = {{View Independent Generative Adversarial Network for Novel View Synthesis}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=hkfLgR3EnDF},
}
Figure