From Semantic Categories to Fixations: A Novel Weakly-Supervised Visual-Auditory Saliency Detection Approach

Guotao Wang, Chenglizhao Chen, Deng-Ping Fan, Aimin Hao, Hong Qin

Abstract

Predicting human visual attention in video sequences, particularly those with accompanying audio, typically requires large datasets annotated with real human eye-tracking data. This paper presents a novel weakly-supervised approach that bypasses the need for expensive human fixations by leveraging simple video category tags. We introduce Selective Class Activation Mapping (SCAM) to generate high-quality "pseudo-fixations," which are then used to train a robust spatial-temporal-audio (STA) network. Our model achieves performance comparable to fully supervised state-of-the-art methods across multiple benchmarks.

Introduction

Visual saliency detection aims to model where humans look when viewing a scene. In the context of video, this involves processing spatial, temporal, and auditory cues. While deep learning has significantly advanced this field, most high-performing models rely on full supervision, requiring massive amounts of human eye-fixation data. Collecting such data is labor-intensive and requires specialized hardware, creating a significant bottleneck for scaling these models.

Existing weakly-supervised methods often rely on Class Activation Mapping (CAM) to identify salient regions. However, standard CAM is designed for object recognition and tends to produce broad, blurry heatmaps that cover entire objects, failing to capture the precise and scattered nature of human fixations. This work addresses these challenges by transforming easily obtainable semantic category tags into accurate saliency predictions.

Methodology

The proposed framework consists of a two-stage pipeline designed to bridge the gap between high-level semantic labels and pixel-level fixation maps.

Overview of the proposed weakly-supervised saliency detection framework.

Selective Class Activation Mapping (SCAM)

The core innovation is the SCAM technique, which improves upon traditional CAM by integrating multi-modal information. We utilize three distinct classification streams:

  • Spatial (S): Processes individual frames.
  • Spatio-temporal (ST): Captures motion cues across multiple frames.
  • Spatial-audio (SA): Integrates visual features with audio spectrum histograms.

A "selective fusion" strategy is employed to combine these maps. Classification confidence scores from each stream act as weights, ensuring that the most reliable modality for a specific frame dominates the final pseudo-fixation map.

Coarse-to-Fine Strategy

To better mimic the precision of human fixations and handle complex backgrounds, we implement a coarse-to-fine strategy. In the "coarse" stage, SCAM identifies a general region of interest. This region is then cropped and passed through a "fine" stage, where SCAM is reapplied to reveal more discriminative local details.

The two-stage SCAM process: Coarse localization followed by fine-grained pseudo-fixation generation.

Audio Switch and STA Network

To mitigate the impact of irrelevant or noisy audio (e.g., background music), an "Audio Switch" module is introduced. It uses an off-the-shelf classifier to verify if the audio's semantic category matches the video's tag; if they diverge, the audio influence is suppressed. Finally, the generated pseudo-fixations serve as ground truth to train a dedicated Spatial-Temporal-Audio (STA) network, which can then predict saliency on unseen videos without requiring category tags.

Experimental Results

The model was trained on the Audio Visual Event (AVE) dataset, using only category tags. Evaluation was performed on six established benchmarks with real human fixations (AVAD, Coutrot1, Coutrot2, DIEM, SumMe, and ETMD).

Performance and Insights

  • Competitive Accuracy: Despite the lack of human-annotated training data, the model performs on par with, and sometimes exceeds, fully supervised state-of-the-art models like DeepNet.
  • Weak Supervision Lead: Our STANet significantly outperforms existing weakly-supervised and unsupervised methods across all test datasets. For instance, on the AVAD dataset, it achieved an AUC-J of 0.873 and a Linear Correlation Coefficient (CC) of 0.438.
  • Ablation Study: The Audio Switch module provided a consistent performance boost of 1.5% to 2%, highlighting the importance of semantic consistency in multi-modal fusion.

Conclusion

This work demonstrates that semantic category labels can serve as effective proxies for human eye fixations. By refining CAM through multi-modal selective fusion and a coarse-to-fine strategy, we can generate high-quality pseudo-fixations that enable robust training of saliency models. This approach significantly reduces the dependency on expensive human-annotated data, paving the way for more scalable visual-auditory attention modeling. Future work will focus on handling multi-labeled videos and exploring more advanced attention mechanisms within the network architecture.

Summary updated

Abstract

Thanks to the rapid advances in the deep learning techniques and the wide availability of large-scale training sets, the performances of video saliency detection models have been improving steadily and significantly. However, the deep learning based visual-audio fixation prediction is still in its infancy. At present, only a few visual-audio sequences have been furnished with real fixations being recorded in the real visual-audio environment. Hence, it would be neither efficiency nor necessary to re-collect real fixations under the same visual-audio circumstance. To address the problem, this paper advocate a novel approach in a weakly-supervised manner to alleviating the demand of large-scale training sets for visual-audio model training. By using the video category tags only, we propose the selective class activation mapping (SCAM), which follows a coarse-to-fine strategy to select the most discriminative regions in the spatial-temporal-audio circumstance. Moreover, these regions exhibit high consistency with the real human-eye fixations, which could subsequently be employed as the pseudo GTs to train a new spatial-temporal-audio (STA) network. Without resorting to any real fixation, the performance of our STA network is comparable to that of the fully supervised ones.
BibTeX
@inproceedings{wang2021from,
  author = {Guotao Wang and Chenglizhao Chen and Deng-Ping Fan and Aimin Hao and Hong Qin},
  title = {{From Semantic Categories to Fixations: A Novel Weakly-Supervised Visual-Auditory Saliency Detection Approach}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.01487},
}
Figure