TL;DR
This paper introduces a novel method for predicting visual saliency in omnidirectional images (OMIs) by adapting existing 2D saliency predictors. They achieve this by processing both the equirectangular projection (ERP) and cube map projections (CMP) of OMIs, and importantly, introduce a sphere-domain optimization step to improve spatial coherence and accuracy. This work is crucial for applications like VR and efficient OMI streaming, where understanding user attention is key.
What Problem Does This Paper Address?
Omnidirectional images (OMIs), commonly used in virtual reality (VR) and immersive applications, present a significant challenge for traditional 2D saliency prediction models. While OMIs can be represented in formats like equirectangular projection (ERP), these are not directly suitable for standard 2D saliency algorithms due to inherent geometric distortions, especially at the poles. To view an OMI, a specific "viewport" is extracted, which requires converting the ERP image to a spherical domain and then projecting it. This means that a saliency map generated directly on the ERP might not accurately reflect where a user will actually look within a chosen viewport. The core problem is how to accurately predict where a human will focus their attention within an OMI, considering its spherical nature and the way users interact with it. This is vital for optimizing bandwidth in streaming, enhancing user experience in VR, and improving object tracking in 360-degree environments.
Methods
The proposed approach combines two main strategies:
- ERP-based Saliency Prediction: They leverage existing 2D saliency predictors (specifically SAM-ResNet [7]) on multiple rotated versions of the ERP image. This accounts for the fact that standard predictors might perform better on certain parts of the ERP (e.g., the equator) due to less distortion. By averaging saliency maps from 8 different longitudinal rotations, they aim to capture a more global saliency.
- CMP-based Saliency Prediction: They convert the OMI into a cube map projection (CMP), which consists of six square faces. Since CMP faces are essentially standard 2D images without the extreme distortions of the ERP poles, existing 2D saliency predictors (specifically BMS [5]) can be applied directly to each face. To handle potential discontinuities at face boundaries, they also consider multiple orientations of the CMP.
Key Innovations:
- Combined ERP and CMP Processing: Instead of relying on just one projection, they process both ERP and CMP to leverage their respective strengths.
- Equator Bias: Recognizing that users tend to focus more on regions near the equator in VR, they introduce a weighted "equator bias" map to refine the combined saliency.
- Sphere Domain Optimization: This is a crucial contribution. After combining the saliency maps from ERP and CMP and applying the equator bias, they perform a smoothing optimization directly on the sphere domain. This ensures that the saliency map is spatially coherent and preserves edges, addressing the limitations of simple averaging or re-projection. This optimization uses a subset of uniformly sampled points on the sphere as seeds for smoothing.
Datasets
The paper primarily uses a public dataset of omnidirectional images [19] for evaluation. They also refer to the ICME 2017 Grand Challenge Salient360! dataset [22] for verification.
Key Insights and Ideas
- No Single Projection is Perfect: The research highlights that neither the ERP nor the CMP alone is sufficient for accurate saliency prediction. Combining insights from both, along with considering multiple orientations, is necessary.
- Geometric Distortion Matters: The paper emphasizes how the geometric distortions inherent in the ERP format directly impact the performance of standard 2D saliency models, necessitating specific handling.
- Sphere-Domain Optimization is Crucial: The most significant insight is the effectiveness of performing optimization directly on the sphere. This allows for spatially consistent saliency maps that better reflect human visual perception in immersive environments, overcoming artifacts from simple re-projection.
- Equator Bias Reflects User Behavior: Incorporating a bias towards the equator aligns with observed human viewing patterns in VR, leading to more realistic saliency predictions.
Key Results
The proposed method achieves competitive results, particularly demonstrating significant improvements in the Normalized Saliency Scanpath (NSS) metric.
- CMP-based Saliency: Using five orientations for CMP-based prediction significantly improves results over a single orientation, as shown by a lower KLD score (0.515 vs. 3.50) and higher CC (0.545 vs. 0.56) for test image P33.
- Combined Saliency: Combining ERP-based and CMP-based saliency maps further improves performance, reducing KLD from 0.55 to 0.39 and increasing CC from 0.56 to 0.69.
- Sphere Optimization Impact: Figure 4 visually demonstrates that the sphere domain optimization enhances the saliency map, with some regions showing increased saliency values consistent with subjective observations. Table 1 shows that for test image P33, optimization with K=100 or K=1000 parameters leads to improved KLD and CC scores, and a substantial boost in NSS (0.96 vs. 0.53 without optimization).
- Comparison with Other Methods: In Table 5, the proposed method outperforms several other approaches in NSS score (1.027 vs. up to 0.971) and achieves comparable performance in KLD and CC metrics on the Salient360! dataset.
Limitations and Future Work
The authors acknowledge several avenues for future work:
- Replacing ERP-based Prediction: They suggest that the ERP-based saliency prediction could be replaced by multi-view based saliency prediction to potentially extract visual attention more effectively.
- Combining with Training-Based Predictors: The current model is largely non-training based (except for the underlying SAM-ResNet). They propose combining their approach with training-based saliency predictors to achieve even better results.
- Further Optimization: While the sphere domain optimization is effective, further refinements could be explored.
Summary updated