360 Degree Panorama Synthesis From Sequential Views Based on Improved FC-Densenets

Dandan Zhu, Qiangqiang Zhou, Tian Han, Yongqing ChenIEEE Access 2019 · OpenReview

Abstract

Deep learning has significantly advanced panorama saliency prediction, yet these models are often hampered by the scarcity of large-scale labeled datasets. To overcome this, we propose a novel generative model designed to synthesize realistic, high-resolution 360° panoramas from sequential views. Our framework employs a Generative Adversarial Network (GAN) featuring an improved Fully-Convolutional Densely Connected Network (FC-DenseNet) as the generator and a specialized correlation layer within the discriminator. This correlation layer enables pixel-level accuracy by calculating similarities between generated and ground-truth images. Experimental results demonstrate that our method outperforms existing baselines, exhibiting superior generalization for real-world data synthesis.

Introduction

360° panoramic images are fundamental to immersive Virtual Reality (VR) environments. However, they present significant challenges: they are high-resolution, difficult to transmit, and expensive to label for tasks like saliency prediction. Existing datasets are often too small to train robust deep learning models effectively. While synthesis offers a potential solution, standard GANs frequently produce blurry results or visible artifacts at the seams where sequential views are stitched. This research introduces a model capable of generating sharp, realistic panoramas from simple sequential views to bridge the data gap in panoramic research.

Methodology

The proposed GAN-based framework introduces several architectural innovations to ensure high-fidelity synthesis.

Local View Extraction

To prepare the training data, 360° images are projected onto a sphere. A virtual camera then "captures" 36 overlapping perspective views. These sequential views serve as the primary input for the synthesis process, allowing the model to learn from manageable, high-resolution segments of the full panorama.

Figure 4: The process of extracting sequential perspective views from a 360° spherical projection.

Generator: Improved FC-DenseNet

The generator utilizes a 56-layer Fully-Convolutional Densely Connected Network. Unlike standard CNNs, the dense connections facilitate better feature reuse and gradient flow. This architecture is crucial for retaining fine-grained textures and details that are typically lost in deeper, traditional networks.

Discriminator and Correlation Layer

A key innovation is the inclusion of a correlation layer in the discriminator. While standard discriminators evaluate the generated image in isolation, this layer explicitly calculates the pixel-level similarity between the generated view and the ground-truth view. This forces the generator to achieve higher precision, resulting in significantly sharper edges.

Weighted Image Patch Loss

To manage complex scenes, the authors implement a weighted loss function. Images are divided into foreground, middle, and background patches, with different weights applied to each. This approach prevents the model from over-simplifying or blurring high-complexity areas, ensuring consistent quality across the entire frame.

Evaluation and Results

The model was evaluated using the Salient360! benchmark dataset and real-world data collected via smartphones.

Quantitative Performance

The proposed method was compared against pix2pix and pix2pixHD. On the Salient360! dataset, our model achieved a Structural Similarity Index (SSIM) of 0.4635, outperforming pix2pixHD (0.3942) and pix2pix (0.3522). It also recorded a superior Peak Signal-to-Noise Ratio (PSNR) of 16.3651. Ablation studies confirmed the importance of the correlation layer, as its removal led to a drop in SSIM from 0.4635 to 0.4257.

Qualitative Analysis

Visual comparisons highlight the model's ability to maintain sharp textures and realistic lighting, whereas baseline models often produced "smoothed" or blurry outputs. Testing on real-world smartphone data further demonstrated the model's robust generalization capabilities outside of controlled benchmark environments.

Conclusion

While the model excels at synthesizing individual high-quality views, seamlessly stitching these views into a perfect 360° sphere remains a challenge. Future work will focus on optimizing the FC-DenseNet architecture for reduced computational intensity and improving the consistency of generated views to eliminate visible seams in VR applications.

Summary updated

Abstract

Inspired by the effectiveness of deep learning model, many panorama saliency prediction models based on deep learning began to emerge and achieved significant performance improvement. However, this kind of model requires a large number of labeled ground-truth data, and the existing panorama datasets are small-scale and difficult to train the deep learning models. To address this problem, we propose a novel panorama generative model for synthesizing realistic and sharp-looking panorama. In particular, our proposed panorama generative model consists of two sub-networks of generator and discriminator. At first, in order to make the synthesized panorama more realistic, we employ the improved Fully-Convolutional Densely Connected Convolutional Networks (FC-DenseNets) as the generator network. Secondly, we design a new correlation layer in the discriminator network, which can calculate the similarity between the generated image and the ground-truth image, and achieve the pixel level accuracy. The experimental results show that our proposed method outperforms other baseline work and has superior generalization ability to synthesize real-world data.
BibTeX
@inproceedings{zhu2019degree,
  author = {Dandan Zhu and Qiangqiang Zhou and Tian Han and Yongqing Chen},
  title = {{360 Degree Panorama Synthesis From Sequential Views Based on Improved FC-Densenets}},
  booktitle = {IEEE Access},
  year = {2019},
  url = {https://openreview.net/forum?id=8E8-ZG0WVP},
}
Figure