Abstract
Synthesizing novel views from a 2D image requires inferring 3D structure and projecting it back to 2D from a new viewpoint. In this paper, we propose an encoder-decoder based generative adversarial network, VI-GAN, to tackle this problem. Our method allows the network, after seeing many images of objects belonging to the same category in different views, to obtain essential knowledge of the intrinsic properties of the objects. To this end, an encoder is designed to extract a view-independent feature that characterizes intrinsic properties of the input image, including 3D structure, color, and texture. We also make the decoder hallucinate the image of a novel view based on the extracted feature and an arbitrary user-specific camera pose. Extensive experiments demonstrate that our model can synthesize high-quality images in different views with continuous camera poses and is general for various applications.
Introduction
Novel view synthesis—generating an image of an object from a new perspective given a single source image—is a classic challenge in computer vision. Traditionally, researchers have approached this through geometry-based methods or learning-based methods. Geometry-based methods attempt to explicitly reconstruct 3D models, which is often ill-posed from a single 2D image and leads to distortions. Learning-based methods use neural networks to map views directly but often struggle with structural integrity and generalization to unseen viewpoints. VI-GAN bridges this gap by learning 3D structure implicitly, understanding an object's 3D essence without requiring explicit 3D supervision like voxels or point clouds during training.
Methodology: The VI-GAN Architecture
The core of the proposed system is the View Independent Generative Adversarial Network (VI-GAN). Its architecture follows an encoder-decoder structure designed to disentangle identity from pose.
Key Components
- The Encoder ($E$): Extracts a "view-independent feature" representing the object's shape, color, and texture, independent of the camera's position.
- The Decoder ($D$): Acts as a neural renderer, taking the view-independent feature and a target camera pose to synthesize the new image. It utilizes Adaptive Instance Normalization (AdaIN) to inject target camera poses, which significantly improves performance over standard concatenation.
- Dual Discriminators: The system employs two discriminators: one for image realism and a Pose Prediction Discriminator to ensure the generated image matches the requested camera angle.
- Loss Functions: Training involves a combination of losses, including a View-Independent Loss to ensure feature consistency across views and a Cycle Consistency Loss to maintain structural integrity during transformations.
Experiments and Results
VI-GAN was evaluated on three major benchmarks: ShapeNet (chairs, sofas, and benches), Multi-PIE (human faces at discrete angles), and 300W-LP (faces with continuous camera poses).
Performance on Complex Objects
On the ShapeNet dataset, VI-GAN achieved the lowest $L_1$ error and highest Structural Similarity Index (SSIM) compared to previous methods like MV3D and Appearance-Flow. Qualitatively, the model excels at "hallucinating" hidden parts of an object, such as the back of a chair, without the blurry artifacts common in earlier learning-based approaches.
Facial Synthesis and Implicit 3D Knowledge
For facial synthesis on the Multi-PIE dataset, VI-GAN achieved a significantly lower FID score (6.51) compared to CR-GAN (8.76), indicating superior image quality. A key insight from these experiments is that the "view-independent feature" implicitly learns 3D geometry; the authors demonstrated that a simple linear model could predict 3D facial landmarks from these features with high accuracy.
Conclusion
VI-GAN demonstrates that forcing a latent space to be view-independent allows a model to effectively separate identity from pose. This disentanglement enables high-quality novel view synthesis with continuous camera movement. While the model currently requires category-specific training and can struggle with extreme occlusions, its ability to implicitly learn 3D structures suggests it could serve as a powerful pre-training step for broader 3D reconstruction tasks.
Summary updated