Deep View Synthesis via Self-Consistent Generative Network

Zhuoman Liu, Wei Jia, Ming Yang, Peiyao Luo, Yong Guo, Mingkui Tan

Introduction

Synthesizing novel camera views from a limited set of source images is a fundamental challenge in computer vision, with applications ranging from immersive video conferencing to virtual reality. Traditional methods often struggle with wide-baseline setups where significant occlusions occur. Researchers have introduced the Self-Consistent Generative Network (SCGN), a deep learning framework designed to synthesize high-quality middle-view images from two wide-baseline side cameras. By leveraging a novel self-consistency constraint, the model achieves state-of-the-art performance in real-time without the need for explicit depth estimation or geometric rectification.

The Challenge of Wide-Baseline Synthesis

View synthesis typically involves creating a "virtual" camera perspective from existing images captured at different angles. Conventional approaches rely heavily on geometry-based techniques, such as calculating depth maps and warping pixels. However, these methods frequently fail when cameras are positioned far apart (a wide baseline). In such scenarios, large portions of the scene visible to one camera may be occluded from the other. This leads to significant artifacts, blurring, or a total failure to establish pixel correspondences. The SCGN approach sidesteps these geometric hurdles by treating view synthesis as a generative task that prioritizes structural and view consistency.

Methodology: The Self-Consistent Generative Network

The SCGN architecture employs a unique "forward-backward" structure to ensure high-fidelity synthesis:

  1. View Synthesis Network (VSN): This forward component uses an encoder-decoder structure to merge features from the left and right input images. To bridge the large spatial gaps inherent in wide-baseline setups, the network utilizes dilated convolutions, which expand the receptive field without increasing the parameter count.
  2. View Decomposition Network (VDN): The backward component is the system's primary innovation. It attempts to reconstruct the original left and right source views from the synthesized middle view.
  3. Self-Consistency Constraint: By requiring the model to successfully "undo" its synthesis, the VDN ensures that the generated middle view preserves all essential information from the source images. This constraint implicitly manages occlusions and maintains the structural integrity of the scene.
  4. Optimized Loss Functions: Training is guided by a combination of Adversarial Loss (to ensure photorealism) and a specialized Sharpness Loss (based on textural complexity) to eliminate the "waxy" artifacts common in generative models.

Experimental Setup and Datasets

The SCGN was rigorously evaluated across three diverse benchmarks:

  • Multi-PIE: A large-scale indoor dataset featuring over 750,000 images of 337 subjects. It was used to test facial synthesis across varying baselines, with angles ranging from $\pm15^\circ$ to $\pm90^\circ$.
  • KITTI: A standard outdoor dataset for autonomous driving, testing the model's robustness in complex, unstructured environments containing elements like foliage and urban infrastructure.
  • Real-world Conferencing Dataset: A custom dataset of 5,000 triplets captured in a dual-camera conferencing environment, featuring diverse subjects and natural motions.

Results and Performance

SCGN demonstrates superior accuracy and robustness compared to existing methods. On the Multi-PIE dataset, it consistently outperformed state-of-the-art models like VSAF and DVM across metrics such as Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (MS-SSIM). Notably, while competing models' performance degraded as the camera baseline increased, SCGN maintained high-quality, recognizable results even at extreme $90^\circ$ angles.

Qualitative comparison of SCGN (Ours) against VSAF and DVM methods. SCGN preserves fine details and structural integrity where others produce ghosting or blur.

Beyond image quality, the model is highly efficient, achieving an inference speed of 27 frames per second (fps) on a single Nvidia TitanX GPU. This makes it a viable solution for real-time applications like live video conferencing.

Limitations and Future Work

While SCGN represents a significant advancement, certain challenges remain. The model is sensitive to rapid motion, which can cause motion blur in source frames that negatively impacts the synthesis. Additionally, it may struggle with out-of-distribution poses or subjects that are not centered between the two cameras, as these scenarios were less frequent in the training data. Future research will focus on integrating temporal consistency across frame sequences to better handle dynamic movement and reduce transient distortions.

Summary updated

BibTeX
@article{liu2022deep,
  author = {Zhuoman Liu and Wei Jia and Ming Yang and Peiyao Luo and Yong Guo and Mingkui Tan},
  title = {{Deep View Synthesis via Self-Consistent Generative Network}},
  journal = {IEEE Trans. Multimedia},
  year = {2022},
  doi = {10.1109/tmm.2021.3053401},
}
Figure