A Spatial-Temporal Dual-Mode Mixed Flow Network for Panoramic Video Salient Object Detection

Xiaolei Chen, Pengcheng Zhang, Zelong Du, Ishfaq AhmadarXiv 2023 · DOI · arXiv

Abstract

Salient Object Detection (SOD) in panoramic (360°) video is a challenging task due to the expansive field of view, high resolution, and the unique spherical geometry of the data. This paper presents STDMMF-Net, a Spatial-Temporal Dual-Mode Mixed Flow Network designed to efficiently identify salient objects by integrating spatial appearance and temporal motion cues. The architecture leverages three novel modules—Inter-Layer Attention (ILA), Inter-Layer Weight (ILW), and Bi-Modal Attention (BMA)—to optimize feature fusion across hierarchical levels and modalities. Experimental results on the SHD360 and ASOD60K datasets demonstrate that STDMMF-Net achieves state-of-the-art performance while significantly reducing computational overhead and memory requirements compared to existing 2D and 3D SOD models.

Introduction

Salient Object Detection (SOD) aims to mimic human visual attention by identifying the most prominent objects in a scene. While traditional 2D video SOD has matured, panoramic video introduces significant hurdles. The wide field of view often results in small salient objects being overwhelmed by massive backgrounds. Furthermore, the high resolution of 360° content makes processing computationally expensive, and standard 2D methods often fail to account for the distortions inherent in equirectangular projections. To address these issues, researchers have developed STDMMF-Net, a lean yet powerful architecture tailored for the complexities of panoramic video.

Proposed Architecture: STDMMF-Net

The Spatial-Temporal Dual-Mode Mixed Flow Network (STDMMF-Net) adopts a dual-stream approach to capture both static appearance and dynamic motion. It processes raw video frames through a Spatial Flow and calculated optical flow through a Time Flow.

The overall architecture of STDMMF-Net, highlighting the spatial and temporal flows and the integration of the three core attention modules.

The network's backbone is based on ResNet34, and its performance is driven by three specialized plug-and-play modules:

  1. Inter-Layer Attention (ILA): This module facilitates communication between adjacent feature levels. By comparing fine-grained spatial details from lower layers with high-level semantic information from deeper layers, ILA helps the network suppress background noise and focus on salient regions.
  2. Inter-Layer Weight (ILW): Recognizing that different hierarchical levels contain varying amounts of salient information, the ILW module dynamically assigns weights to each of the five feature levels. This ensures that the most informative features contribute most heavily to the final prediction.
  3. Bi-Modal Attention (BMA): This module is responsible for the final fusion of spatial and temporal features. It ensures that the model effectively combines "what" an object looks like with "how" it moves.

Key Innovations and Insights

The Importance of Motion

A critical finding of this research is the dominance of motion in driving attention within 360° environments. The "Mixed Flow" (combining spatial and temporal data) outperformed spatial-only models by up to 15% in Mean Absolute Error (MAE). This suggests that in wide-angle panoramic scenes, movement is often the primary cue that draws a viewer's eye.

Inter-Layer Weighting for Efficiency

To maintain a low computational footprint, the ILW module quantifies the saliency contribution of each layer. By intelligently weighting these features before fusion, the network avoids the redundancy typical of standard multi-level fusion strategies.

Detailed structure of the Interlayer Weight (ILW) module, showing how weights are calculated for both spatial and temporal feature sets.

The Depth Map Dilemma

Interestingly, the researchers found that incorporating depth maps—a common technique in 2D SOD—actually degraded performance. This is attributed to the lack of depth estimation models specifically optimized for the spherical geometry of panoramic video, which often produce inaccurate structural cues that mislead the network.

Experimental Results

STDMMF-Net was evaluated against seven state-of-the-art methods on the SHD360 and ASOD60K datasets.

  • Performance: On the ASOD60K dataset, the model achieved a 22.41% improvement in MAE and a 2.49% increase in F-measure over the previous best model.
  • Efficiency: The model is significantly more efficient than its competitors, reducing reasoning memory by up to 205% and drastically cutting testing time, making it suitable for real-time VR/AR applications.
  • Generalization: The model showed robust performance when trained on one dataset and tested on another, indicating that it learns generalizable salient features rather than dataset-specific biases.

Conclusion and Future Work

STDMMF-Net represents a significant step forward in panoramic video SOD, offering a high-accuracy, low-complexity solution. Future work will focus on developing specialized depth estimation algorithms for 360° video to successfully integrate structural cues. Additionally, the researchers plan to explore multi-modal inputs, such as audio or user head-tracking data, to further refine saliency predictions in immersive environments.

Summary updated

Abstract

Salient object detection (SOD) in panoramic video is still in the initial exploration stage. The indirect application of 2D video SOD method to the detection of salient objects in panoramic video has many unmet challenges, such as low detection accuracy, high model complexity, and poor generalization performance. To overcome these hurdles, we design an Inter-Layer Attention (ILA) module, an Inter-Layer weight (ILW) module, and a Bi-Modal Attention (BMA) module. Based on these modules, we propose a Spatial-Temporal Dual-Mode Mixed Flow Network (STDMMF-Net) that exploits the spatial flow of panoramic video and the corresponding optical flow for SOD. First, the ILA module calculates the attention between adjacent level features of consecutive frames of panoramic video to improve the accuracy of extracting salient object features from the spatial flow. Then, the ILW module quantifies the salient object information contained in the features of each level to improve the fusion efficiency of the features of each level in the mixed flow. Finally, the BMA module improves the detection accuracy of STDMMF-Net. A large number of subjective and objective experimental results testify that the proposed method demonstrates better detection accuracy than the state-of-the-art (SOTA) methods. Moreover, the comprehensive performance of the proposed method is better in terms of memory required for model inference, testing time, complexity, and generalization performance.
BibTeX
@misc{chen2023spatial,
  author = {Xiaolei Chen and Pengcheng Zhang and Zelong Du and Ishfaq Ahmad},
  title = {{A Spatial-Temporal Dual-Mode Mixed Flow Network for Panoramic Video Salient Object Detection}},
  year = {2023},
  eprint = {2310.09016},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.09016},
}
Figure