Deep View Synthesis via Self-Consistent Generative Network

Zhuoman Liu, Wei Jia, Ming Yang, Peiyao Luo, Yong Guo, Mingkui Tan

TL;DR

Researchers have developed the Self-Consistent Generative Network (SCGN), a deep learning model that synthesizes high-quality middle-view images from two wide-baseline side cameras. By enforcing a "self-consistency" constraint—where the model must be able to decompose its synthesized view back into the original inputs—it achieves state-of-the-art results in real-time without requiring explicit depth maps or geometric rectification.

What Problem Does This Paper Address?

View synthesis is the art of creating a "virtual" camera view from existing images taken at different angles. It’s a cornerstone for technologies like video conferencing, VR, and free-viewpoint TV.

The traditional approach relies on geometry-based methods: calculating depth and warping pixels from one view to another. However, this breaks down when cameras have a large baseline (they are far apart). In these scenarios, "occlusion" becomes a nightmare—parts of the scene visible to one camera are hidden from the other. Geometry-based models often produce blurry artifacts or fail to find pixel correspondences entirely.

The authors aim to bridge this gap by treating view synthesis as a generative task that preserves "view consistency" without needing to explicitly solve for complex 3D geometry.

Methods

The core of the paper is the Self-Consistent Generative Network (SCGN), which uses a clever "forward-backward" architecture:

  1. View Synthesis Network (VSN): This is the "forward" part. An encoder-decoder network takes the left and right images and merges their features to generate a synthesized middle view. To handle the large spatial gaps between cameras, the authors use dilated convolutions, which expand the network's "field of vision" without adding extra parameters.
  2. View Decomposition Network (VDN): This is the "backward" part and the paper's key innovation. It takes the synthesized middle view and attempts to reconstruct the original left and right input views.
  3. Self-Consistency Constraint: By forcing the model to be able to "undo" its synthesis, the VDN ensures that the synthesized image contains all the necessary information from the source views. This implicitly handles occlusions and preserves structural integrity.
  4. Multi-faceted Loss Functions: The model isn't just trained on pixel accuracy. It uses an Adversarial Loss (GAN-style) to ensure photo-realism and a specialized Sharpness Loss (based on textural complexity) to prevent the "waxy" or blurry look common in generative models.

Datasets

The authors evaluated SCGN on three distinct benchmarks:

  • Multi-PIE: A massive indoor dataset of 750,000+ images of 337 people captured from 15 different viewpoints. This is used to test facial synthesis across various "baselines" (angles ranging from $\pm15^\circ$ to $\pm90^\circ$).
  • KITTI: A standard outdoor dataset for autonomous driving. It tests the model’s ability to handle complex, unstructured environments like streets and trees.
  • Real-world Conferencing Dataset: A new dataset collected by the authors (roughly 5,000 triplets) specifically for a dual-camera video conferencing setup, featuring people with different clothing and motions.

Key Insights and Ideas

  • Geometry-Free is a Feature: By avoiding explicit geometric rectification (like image warping), the model is much easier to train and deploy. It learns the "mapping" between views directly from the data.
  • Self-Consistency $\neq$ Cycle-Consistency: While similar to the popular CycleGAN approach, SCGN’s self-consistency is asymmetric. It maps two domains (side views) into one (middle view) and then decomposes them back. This specific structure is what allows it to "fill in" occluded areas effectively.
  • Dilated Convolutions Matter: The authors found that replacing standard convolutional layers with dilated ones significantly improved the model's ability to "reach across" the large distance between wide-baseline cameras.

Key Results

  • Superior Accuracy: On the Multi-PIE dataset, SCGN consistently outperformed state-of-the-art methods (like VSAF and DVM) across all metrics, including PSNR (Peak Signal-to-Noise Ratio) and MS-SSIM (Structural Similarity).
  • Robustness to Wide Angles: As the camera baseline increased (e.g., from $15^\circ$ to $90^\circ$), competing models' performance plummeted, while SCGN maintained high-quality, recognizable results.
  • Real-Time Performance: The model achieves an inference speed of 27 frames per second (fps) on a single Nvidia TitanX GPU, making it viable for real-time video conferencing applications.
  • Qualitative Edge: Visual comparisons show that SCGN preserves fine details—like the texture of a road or the specific features of a face—where other models produce "ghosting" or severe blurring.

Limitations and Future Work

Despite its strengths, the authors acknowledge several hurdles:

  • Rapid Motion: The model struggles with "motion blur" in the input images. If the source frames are blurry due to fast movement, the synthesis fails.
  • Out-of-Distribution Poses: The model was primarily trained on upper bodies. Consequently, it struggles to synthesize accurate hand movements or extremely "large poses" (like someone leaning far out of frame) because those examples were rare in the training data.
  • Non-Centered Subjects: If a person is not centered between the two cameras, the face area can still suffer from minor distortions.

Future work likely involves integrating temporal consistency (looking at sequences of frames rather than just pairs) to better handle the "motion blur" and "rapid movement" issues.

Summary updated

Abstract

View synthesis aims to produce unseen views from a set of views captured by two or more cameras at different positions. This task is non-trivial since it is hard to conduct pixel-level matching among different views. To address this issue, most existing methods seek to exploit the geometric information to match pixels. However, when the distinct cameras have a large baseline (i. e., far away from each other), severe geometry distortion issues would occur and the geometric information may fail to provide useful guidance, resulting in very blurry synthesized images. To address the above issues, in this paper, we propose a novel deep generative model, called Self-Consistent Generative Network (SCGN), which synthesizes novel views from the given input views without explicitly exploiting the geometric information. The proposed SCGN model consists of two main components, i. e., a View Synthesis Network (VSN) and a View Decomposition Network (VDN), both employing an Encoder-Decoder structure. Here, the VDN seeks to reconstruct input views from the synthesized novel view to preserve the consistency of view synthesis. Thanks to VDN, SCGN is able to synthesize novel views without using any geometric rectification before encoding, making it easier for both training and applications. Finally, adversarial loss is introduced to improve the photo-realism of novel views. Both qualitative and quantitative comparisons against several state-of-the-art methods on two benchmark tasks demonstrated the superiority of our approach.
BibTeX
@article{liu2022deep,
  author = {Zhuoman Liu and Wei Jia and Ming Yang and Peiyao Luo and Yong Guo and Mingkui Tan},
  title = {{Deep View Synthesis via Self-Consistent Generative Network}},
  journal = {IEEE Trans. Multimedia},
  year = {2022},
  doi = {10.1109/tmm.2021.3053401},
}
Figure