Abstract
Multi-level features are critical for saliency detection. Effectively combining these features with temporal information can significantly enhance the accuracy of video saliency models. We propose TSFP-Net, a 3D fully convolutional encoder-decoder architecture that integrates scale, space, and time information. The encoder extracts multi-scale temporal-spatial features, which are then structured into a feature pyramid via top-down integration. A hierarchical decoder processes these features to produce final saliency maps. Our model is efficient, capable of real-time performance, and achieves state-of-the-art results across multiple visual and audio-video saliency benchmarks.
Introduction
Predicting human visual attention in video—video saliency detection—is essential for applications ranging from smart video compression to automated captioning. Humans simultaneously attend to large, semantic objects and small, fast-moving details. Traditional models often employ a "2D CNN + LSTM" approach, which suffers from several drawbacks:
- Transfer Learning Gaps: Pre-training on static images often fails to account for the resolution and shape differences inherent in video datasets.
- LSTM Bottlenecks: Sequential processing in LSTMs creates discrepancies between training (where initial states are often zeroed) and testing (where states are continuous).
- Temporal Overlay: LSTMs treat time as a layer on top of spatial data rather than processing spatio-temporal information as a unified 3D volume.
TSFP-Net addresses these issues by utilizing 3D convolutions and a well-designed feature pyramid to capture multi-level features across the temporal dimension.
TSFP-Net Architecture
The architecture of TSFP-Net is designed to be simple yet effective, focusing on the integration of multi-scale features.
3D Encoder and Feature Pyramid
The model uses an S3D (Separable 3D CNN) backbone as its encoder. S3D decomposes standard 3D kernels into separate spatial and temporal filters, enhancing efficiency without sacrificing representational power. The core innovation is the Temporal-Spatial Feature Pyramid. By passing deep semantic features back to shallower layers through a top-down path, the network ensures that high-level "what" information and low-level "where" (motion) information are fused effectively.
Hierarchical Decoder
Instead of relying solely on the final layer, TSFP-Net performs independent decoding at multiple scales. These multi-scale outputs are integrated to produce the final saliency map, ensuring that both large salient regions and minute, fast-moving points are accurately captured.
Audio Integration
For audio-visual tasks, the model incorporates a SoundNet branch. Audio features are extracted and fused with visual features using a bilinear transformation and an attention mechanism, allowing auditory cues to guide visual attention.
Experimental Results
The authors evaluated TSFP-Net on nine benchmarks, including purely visual (DHF1K, Hollywood-2, UCF-sports) and audio-visual (DIEM, Coutrot1/2, etc.) datasets.
Performance on DHF1K
On the DHF1K test set, the largest and most diverse benchmark, TSFP-Net established a new state-of-the-art.
As shown in the table above, TSFP-Net achieves an NSS of 2.966, outperforming previous leaders like ViNet and significantly surpassing traditional LSTM-based architectures.
Efficiency and Real-time Inference
Beyond accuracy, TSFP-Net is highly efficient. With a model size of only 58.4 MB, it is significantly leaner than competitors like DeepVS (344 MB). On an NVIDIA RTX 2080 Ti, it achieves an inference speed of approximately 90 FPS (0.011s per frame), making it ideal for real-time deployment.
Conclusion
TSFP-Net demonstrates that 3D convolutions combined with a temporal-spatial feature pyramid provide a superior framework for video saliency detection. By avoiding the sequential bottlenecks of LSTMs and holistically processing 3D video volumes, the model achieves state-of-the-art performance with remarkable efficiency. While audio integration provided marginal gains on current datasets, future work focusing on audio-dominant scenarios may further unlock the potential of multi-modal saliency detection.
Reference: Chang, Q., & Zhu, S. (2021). Temporal-Spatial Feature Pyramid for Video Saliency Detection. arXiv:2105.04213v2 [cs.CV].
Summary updated