Abstract
Deep learning has significantly advanced panorama saliency prediction, yet these models are often hampered by the scarcity of large-scale labeled datasets. To overcome this, we propose a novel generative model designed to synthesize realistic, high-resolution 360° panoramas from sequential views. Our framework employs a Generative Adversarial Network (GAN) featuring an improved Fully-Convolutional Densely Connected Network (FC-DenseNet) as the generator and a specialized correlation layer within the discriminator. This correlation layer enables pixel-level accuracy by calculating similarities between generated and ground-truth images. Experimental results demonstrate that our method outperforms existing baselines, exhibiting superior generalization for real-world data synthesis.
Introduction
360° panoramic images are fundamental to immersive Virtual Reality (VR) environments. However, they present significant challenges: they are high-resolution, difficult to transmit, and expensive to label for tasks like saliency prediction. Existing datasets are often too small to train robust deep learning models effectively. While synthesis offers a potential solution, standard GANs frequently produce blurry results or visible artifacts at the seams where sequential views are stitched. This research introduces a model capable of generating sharp, realistic panoramas from simple sequential views to bridge the data gap in panoramic research.
Methodology
The proposed GAN-based framework introduces several architectural innovations to ensure high-fidelity synthesis.
Local View Extraction
To prepare the training data, 360° images are projected onto a sphere. A virtual camera then "captures" 36 overlapping perspective views. These sequential views serve as the primary input for the synthesis process, allowing the model to learn from manageable, high-resolution segments of the full panorama.
Generator: Improved FC-DenseNet
The generator utilizes a 56-layer Fully-Convolutional Densely Connected Network. Unlike standard CNNs, the dense connections facilitate better feature reuse and gradient flow. This architecture is crucial for retaining fine-grained textures and details that are typically lost in deeper, traditional networks.
Discriminator and Correlation Layer
A key innovation is the inclusion of a correlation layer in the discriminator. While standard discriminators evaluate the generated image in isolation, this layer explicitly calculates the pixel-level similarity between the generated view and the ground-truth view. This forces the generator to achieve higher precision, resulting in significantly sharper edges.
Weighted Image Patch Loss
To manage complex scenes, the authors implement a weighted loss function. Images are divided into foreground, middle, and background patches, with different weights applied to each. This approach prevents the model from over-simplifying or blurring high-complexity areas, ensuring consistent quality across the entire frame.
Evaluation and Results
The model was evaluated using the Salient360! benchmark dataset and real-world data collected via smartphones.
Quantitative Performance
The proposed method was compared against pix2pix and pix2pixHD. On the Salient360! dataset, our model achieved a Structural Similarity Index (SSIM) of 0.4635, outperforming pix2pixHD (0.3942) and pix2pix (0.3522). It also recorded a superior Peak Signal-to-Noise Ratio (PSNR) of 16.3651. Ablation studies confirmed the importance of the correlation layer, as its removal led to a drop in SSIM from 0.4635 to 0.4257.
Qualitative Analysis
Visual comparisons highlight the model's ability to maintain sharp textures and realistic lighting, whereas baseline models often produced "smoothed" or blurry outputs. Testing on real-world smartphone data further demonstrated the model's robust generalization capabilities outside of controlled benchmark environments.
Conclusion
While the model excels at synthesizing individual high-quality views, seamlessly stitching these views into a perfect 360° sphere remains a challenge. Future work will focus on optimizing the FC-DenseNet architecture for reduced computational intensity and improving the consistency of generated views to eliminate visible seams in VR applications.
Summary updated