Saliency-based Multiple Region of Interest Detection from a Single 360° image

Yuuki Sawabe, Satoshi Ikehata, Kiyoharu Aizawa

Introduction

Omnidirectional (360°) images provide a comprehensive view of a scene, capturing visual information in every direction. However, this wealth of data presents a challenge: the human field of view (FoV) is significantly smaller than the full spherical projection. Consequently, users often overlook important details located outside their immediate line of sight. While previous research has explored saliency in 360° videos—leveraging temporal cues to predict viewer attention—identifying multiple Regions of Interest (RoIs) in a single static 360° image remains a complex task.

This paper presents a framework to automatically detect and extract the most "impressive" regions from a single 360° image. The approach combines a deep-learning-based saliency model with a novel optimization strategy to ensure that the selected regions are both visually interesting and spatially diverse.

Methodology

Spherical Data Augmentation

A significant hurdle in training saliency models for 360° content is "center bias." Because photographers typically align subjects with the horizon (the equator), existing datasets are heavily biased toward the center of the equirectangular projection. Models trained on such data often fail to recognize salient objects near the poles.

To address this, the authors introduced Spherical Random Rotation. Unlike standard 2D data augmentation (e.g., horizontal flipping), this method involves rotating the image data on a virtual 3D sphere before projecting it back into a 2D map. This process forces the model to learn the characteristics of salient objects regardless of their spherical coordinates, significantly improving the robustness of the saliency prediction.

Candidate Generation and Salient-IoU Optimization

The system generates potential RoIs using Selective Search, a standard computer vision algorithm, which produces hundreds of candidate bounding boxes. These are filtered to ensure they correspond to a natural human FoV.

The core of the selection process is the Salient-IoU optimization. Simply selecting the top $N$ most salient regions often results in redundant views of the same object. To prevent this, the authors developed a cost function that balances two metrics:

  1. Saliency: The total visual interest contained within a candidate box.
  2. Diversity (IoU): The Intersection-Over-Union (IoU) between regions.

By applying a greedy algorithm to minimize this cost function, the system selects a set of RoIs that are individually salient but collectively diverse, providing a broad summary of the entire 360° scene.

Experimental Results

Quantitative Performance

The impact of spherical augmentation was evaluated using the Salient360! benchmark. The inclusion of spherical random rotation improved the Pearson’s Correlation Coefficient (CC)—a metric measuring the alignment between predicted saliency and human gaze—from 0.432 to 0.584. This substantial improvement demonstrates the necessity of accounting for spherical geometry in 360° image processing.

Subjective Evaluation

To evaluate the quality of the multi-RoI selection, the authors created a new dataset of 45 outdoor 360° images annotated by 14 workers via Amazon Mechanical Turk. In a head-to-head subjective comparison, users were asked to choose the "most impressive" set of regions. The proposed method’s RoIs were selected 58% of the time, outperforming human-annotated regions, which were preferred in only 27% of trials.

The researchers also determined that a small weight on the IoU penalty ($a=0.03$) provided the optimal balance, confirming that while diversity is essential for a good summary, saliency remains the primary indicator of a high-quality RoI.

Conclusion and Future Work

The proposed framework successfully automates the summarization of 360° images into high-quality, manageable views. By addressing center bias through spherical rotation and ensuring diversity via Salient-IoU optimization, the model provides a superior user experience compared to traditional methods.

Future research will focus on improving the detection of small salient objects and extending the framework to 360° video. In video applications, maintaining temporal consistency—ensuring RoIs do not jump erratically between frames—will be a primary challenge. Additionally, the authors plan to explore optimal interface designs for presenting these multiple RoIs to users in Virtual Reality (VR) environments.

Summary updated

Abstract

360{\deg} images are informative -- it contains omnidirectional visual information around the camera. However, the areas that cover a 360{\deg} image is much larger than the human's field of view, therefore important information in different view directions is easily overlooked. To tackle this issue, we propose a method for predicting the optimal set of Region of Interest (RoI) from a single 360{\deg} image using the visual saliency as a clue. To deal with the scarce, strongly biased training data of existing single 360{\deg} image saliency prediction dataset, we also propose a data augmentation method based on the spherical random data rotation. From the predicted saliency map and redundant candidate regions, we obtain the optimal set of RoIs considering both the saliency within a region and the Interaction-Over-Union (IoU) between regions. We conduct the subjective evaluation to show that the proposed method can select regions that properly summarize the input 360{\deg} image.
BibTeX
@inproceedings{sawabe2022saliency,
  author = {Yuuki Sawabe and Satoshi Ikehata and Kiyoharu Aizawa},
  title = {{Saliency-based Multiple Region of Interest Detection from a Single 360° image}},
  booktitle = {IEEE Access},
  year = {2022},
  url = {https://openreview.net/forum?id=eYCowfdTg_G},
}

Loading related content...

Figure