Temporal-Spatial Feature Pyramid for Video Saliency Detection

Qinyao Chang, Shiping Zhu, Lanyun ZhuarXiv 2021 · OpenReview

Abstract

Multi-level features are critical for saliency detection. Effectively combining these features with temporal information can significantly enhance the accuracy of video saliency models. We propose TSFP-Net, a 3D fully convolutional encoder-decoder architecture that integrates scale, space, and time information. The encoder extracts multi-scale temporal-spatial features, which are then structured into a feature pyramid via top-down integration. A hierarchical decoder processes these features to produce final saliency maps. Our model is efficient, capable of real-time performance, and achieves state-of-the-art results across multiple visual and audio-video saliency benchmarks.

Introduction

Predicting human visual attention in video—video saliency detection—is essential for applications ranging from smart video compression to automated captioning. Humans simultaneously attend to large, semantic objects and small, fast-moving details. Traditional models often employ a "2D CNN + LSTM" approach, which suffers from several drawbacks:

  1. Transfer Learning Gaps: Pre-training on static images often fails to account for the resolution and shape differences inherent in video datasets.
  2. LSTM Bottlenecks: Sequential processing in LSTMs creates discrepancies between training (where initial states are often zeroed) and testing (where states are continuous).
  3. Temporal Overlay: LSTMs treat time as a layer on top of spatial data rather than processing spatio-temporal information as a unified 3D volume.

TSFP-Net addresses these issues by utilizing 3D convolutions and a well-designed feature pyramid to capture multi-level features across the temporal dimension.

TSFP-Net Architecture

The architecture of TSFP-Net is designed to be simple yet effective, focusing on the integration of multi-scale features.

The TSFP-Net architecture featuring the 3D encoder, temporal-spatial feature pyramid, and hierarchical decoder, alongside an optional audio branch.

3D Encoder and Feature Pyramid

The model uses an S3D (Separable 3D CNN) backbone as its encoder. S3D decomposes standard 3D kernels into separate spatial and temporal filters, enhancing efficiency without sacrificing representational power. The core innovation is the Temporal-Spatial Feature Pyramid. By passing deep semantic features back to shallower layers through a top-down path, the network ensures that high-level "what" information and low-level "where" (motion) information are fused effectively.

Hierarchical Decoder

Instead of relying solely on the final layer, TSFP-Net performs independent decoding at multiple scales. These multi-scale outputs are integrated to produce the final saliency map, ensuring that both large salient regions and minute, fast-moving points are accurately captured.

Audio Integration

For audio-visual tasks, the model incorporates a SoundNet branch. Audio features are extracted and fused with visual features using a bilinear transformation and an attention mechanism, allowing auditory cues to guide visual attention.

Experimental Results

The authors evaluated TSFP-Net on nine benchmarks, including purely visual (DHF1K, Hollywood-2, UCF-sports) and audio-visual (DIEM, Coutrot1/2, etc.) datasets.

Performance on DHF1K

On the DHF1K test set, the largest and most diverse benchmark, TSFP-Net established a new state-of-the-art.

Comparison of TSFP-Net against state-of-the-art methods on the DHF1K dataset. Red indicates the best performance, blue indicates the second best.

As shown in the table above, TSFP-Net achieves an NSS of 2.966, outperforming previous leaders like ViNet and significantly surpassing traditional LSTM-based architectures.

Efficiency and Real-time Inference

Beyond accuracy, TSFP-Net is highly efficient. With a model size of only 58.4 MB, it is significantly leaner than competitors like DeepVS (344 MB). On an NVIDIA RTX 2080 Ti, it achieves an inference speed of approximately 90 FPS (0.011s per frame), making it ideal for real-time deployment.

Conclusion

TSFP-Net demonstrates that 3D convolutions combined with a temporal-spatial feature pyramid provide a superior framework for video saliency detection. By avoiding the sequential bottlenecks of LSTMs and holistically processing 3D video volumes, the model achieves state-of-the-art performance with remarkable efficiency. While audio integration provided marginal gains on current datasets, future work focusing on audio-dominant scenarios may further unlock the potential of multi-modal saliency detection.

Reference: Chang, Q., & Zhu, S. (2021). Temporal-Spatial Feature Pyramid for Video Saliency Detection. arXiv:2105.04213v2 [cs.CV].

Summary updated

Abstract

Multi-level features are important for saliency detection. Better combination and use of multi-level features with time information can greatly improve the accuracy of the video saliency model. In order to fully combine multi-level features and make it serve the video saliency model, we propose a 3D fully convolutional encoder-decoder architecture for video saliency detection, which combines scale, space and time information for video saliency modeling. The encoder extracts multi-scale temporal-spatial features from the input continuous video frames, and then constructs temporal-spatial feature pyramid through temporal-spatial convolution and top-down feature integration. The decoder performs hierarchical decoding of temporal-spatial features from different scales, and finally produces a saliency map from the integration of multiple video frames. Our model is simple yet effective, and can run in real time. We perform abundant experiments, and the results indicate that the well-designed structure can improve the precision of video saliency detection significantly. Experimental results on three purely visual video saliency benchmarks and six audio-video saliency benchmarks demonstrate that our method outperforms the existing state-of-the-art methods.
BibTeX
@misc{chang2021temporal,
  author = {Qinyao Chang and Shiping Zhu and Lanyun Zhu},
  title = {{Temporal-Spatial Feature Pyramid for Video Saliency Detection}},
  year = {2021},
  eprint = {2105.04213},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.04213},
}
Figure