Saliency Prediction for Omnidirectional Images Considering Optimization on Sphere Domain

Bhishma Dedhia, Jui-Chiu Chiang, Yi-Fan CharICASSP 2019 · OpenReview

TL;DR

This paper introduces a novel method for predicting visual saliency in omnidirectional images (OMIs) by adapting existing 2D saliency predictors. They achieve this by processing both the equirectangular projection (ERP) and cube map projections (CMP) of OMIs, and importantly, introduce a sphere-domain optimization step to improve spatial coherence and accuracy. This work is crucial for applications like VR and efficient OMI streaming, where understanding user attention is key.

What Problem Does This Paper Address?

Omnidirectional images (OMIs), commonly used in virtual reality (VR) and immersive applications, present a significant challenge for traditional 2D saliency prediction models. While OMIs can be represented in formats like equirectangular projection (ERP), these are not directly suitable for standard 2D saliency algorithms due to inherent geometric distortions, especially at the poles. To view an OMI, a specific "viewport" is extracted, which requires converting the ERP image to a spherical domain and then projecting it. This means that a saliency map generated directly on the ERP might not accurately reflect where a user will actually look within a chosen viewport. The core problem is how to accurately predict where a human will focus their attention within an OMI, considering its spherical nature and the way users interact with it. This is vital for optimizing bandwidth in streaming, enhancing user experience in VR, and improving object tracking in 360-degree environments.

Methods

The proposed approach combines two main strategies:

  1. ERP-based Saliency Prediction: They leverage existing 2D saliency predictors (specifically SAM-ResNet [7]) on multiple rotated versions of the ERP image. This accounts for the fact that standard predictors might perform better on certain parts of the ERP (e.g., the equator) due to less distortion. By averaging saliency maps from 8 different longitudinal rotations, they aim to capture a more global saliency.
  2. CMP-based Saliency Prediction: They convert the OMI into a cube map projection (CMP), which consists of six square faces. Since CMP faces are essentially standard 2D images without the extreme distortions of the ERP poles, existing 2D saliency predictors (specifically BMS [5]) can be applied directly to each face. To handle potential discontinuities at face boundaries, they also consider multiple orientations of the CMP.

Key Innovations:

  • Combined ERP and CMP Processing: Instead of relying on just one projection, they process both ERP and CMP to leverage their respective strengths.
  • Equator Bias: Recognizing that users tend to focus more on regions near the equator in VR, they introduce a weighted "equator bias" map to refine the combined saliency.
  • Sphere Domain Optimization: This is a crucial contribution. After combining the saliency maps from ERP and CMP and applying the equator bias, they perform a smoothing optimization directly on the sphere domain. This ensures that the saliency map is spatially coherent and preserves edges, addressing the limitations of simple averaging or re-projection. This optimization uses a subset of uniformly sampled points on the sphere as seeds for smoothing.

Datasets

The paper primarily uses a public dataset of omnidirectional images [19] for evaluation. They also refer to the ICME 2017 Grand Challenge Salient360! dataset [22] for verification.

Key Insights and Ideas

  • No Single Projection is Perfect: The research highlights that neither the ERP nor the CMP alone is sufficient for accurate saliency prediction. Combining insights from both, along with considering multiple orientations, is necessary.
  • Geometric Distortion Matters: The paper emphasizes how the geometric distortions inherent in the ERP format directly impact the performance of standard 2D saliency models, necessitating specific handling.
  • Sphere-Domain Optimization is Crucial: The most significant insight is the effectiveness of performing optimization directly on the sphere. This allows for spatially consistent saliency maps that better reflect human visual perception in immersive environments, overcoming artifacts from simple re-projection.
  • Equator Bias Reflects User Behavior: Incorporating a bias towards the equator aligns with observed human viewing patterns in VR, leading to more realistic saliency predictions.

Key Results

The proposed method achieves competitive results, particularly demonstrating significant improvements in the Normalized Saliency Scanpath (NSS) metric.

  • CMP-based Saliency: Using five orientations for CMP-based prediction significantly improves results over a single orientation, as shown by a lower KLD score (0.515 vs. 3.50) and higher CC (0.545 vs. 0.56) for test image P33.
  • Combined Saliency: Combining ERP-based and CMP-based saliency maps further improves performance, reducing KLD from 0.55 to 0.39 and increasing CC from 0.56 to 0.69.
  • Sphere Optimization Impact: Figure 4 visually demonstrates that the sphere domain optimization enhances the saliency map, with some regions showing increased saliency values consistent with subjective observations. Table 1 shows that for test image P33, optimization with K=100 or K=1000 parameters leads to improved KLD and CC scores, and a substantial boost in NSS (0.96 vs. 0.53 without optimization).
  • Comparison with Other Methods: In Table 5, the proposed method outperforms several other approaches in NSS score (1.027 vs. up to 0.971) and achieves comparable performance in KLD and CC metrics on the Salient360! dataset.

Limitations and Future Work

The authors acknowledge several avenues for future work:

  • Replacing ERP-based Prediction: They suggest that the ERP-based saliency prediction could be replaced by multi-view based saliency prediction to potentially extract visual attention more effectively.
  • Combining with Training-Based Predictors: The current model is largely non-training based (except for the underlying SAM-ResNet). They propose combining their approach with training-based saliency predictors to achieve even better results.
  • Further Optimization: While the sphere domain optimization is effective, further refinements could be explored.

Summary updated

Abstract

There are several formats to describe the omnidirectional images. Among them, equirectangular projection (ERP), represented as 2D image, is the most widely used format. There exist many outstanding methods capable of well predicting the saliency maps for the conventional 2D images. But these works cannot be directly extended to predict the saliency map of the ERP image, since the content on ERP is not for direct display. Instead, the viewport image on demand is generated after converting the ERP image to the sphere domain, followed by rectilinear projection. In this paper, we propose a model to predict the saliency maps of the ERP images using existing saliency predictors for the 2D image. Some pre-processing and post-processing are used to manage the problem mentioned above. In particular, a smoothing based optimization is realized on the sphere domain. A public dataset of omnidirectional images is used to perform all the experiments and competitive results are achieved.
BibTeX
@inproceedings{dedhia2019saliency,
  author = {Bhishma Dedhia and Jui-Chiu Chiang and Yi-Fan Char},
  title = {{Saliency Prediction for Omnidirectional Images Considering Optimization on Sphere Domain}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=eJtX_1DhvxA},
}
Figure