Weakly Supervised Visual-Auditory Saliency Detection with Multigranularity Perception

Guotao Wang, Chenglizhao Chen, Deng-Ping Fan, Aimin Hao, Hong QinarXiv 2021 · arXiv

Abstract

Predicting human eye fixations in videos with audio, known as Video Fixation Prediction (VFP), typically requires expensive and labor-intensive eye-tracking data. This paper introduces a novel weakly supervised framework that leverages simple video category tags (e.g., "horse," "violin") to generate high-quality "pseudofixations." By employing Selective Class Activation Mapping (SCAM) and a multigranularity perception module (SCAM+), the framework achieves performance comparable to state-of-the-art fully supervised models without the need for real human-eye fixation data during training.

Introduction

Video Fixation Prediction (VFP) aims to predict the scattered coordinates of human gaze patterns, a task significantly more complex than standard Video Salient Object Detection (VSOD), which focuses on segmenting entire objects with clear boundaries. The primary challenge in VFP is the scarcity of large-scale datasets containing real human fixations, especially in visual-audio environments, due to the high cost of specialized eye-tracking hardware and human subjects.

This research bridges the gap by using easily obtainable video category tags as a proxy for supervision. The core hypothesis is that the regions a machine uses to classify a video often align with the regions that attract human attention.

Methodology

Overview of the Framework

The proposed approach moves from simple semantic labels to complex fixation predictions through a multi-stage pipeline involving pseudofixation generation and knowledge distillation.

Overview of the framework: from video tags to predicted fixations via SCAM and knowledge distillation.

Selective Class Activation Mapping (SCAM)

SCAM identifies discriminative regions by decoupling video data into three streams: Spatial (S), Spatiotemporal (ST), and Spatial-Audio (SA). These streams are selectively fused based on classification confidence to generate initial fixation maps.

SCAM+ and Multigranularity Perception

To mimic the global nature of human attention, the authors developed SCAM+, which incorporates multigranularity perception. This module uses a Conservative Graph Convolutional Network (C-GCN) to reason across different temporal and semantic scales:

  • Short-term memory: Neighboring frames.
  • Long-term memory: Distant frames within the same video.
  • Cross-term memory: Frames from different videos sharing the same category tag.
Rationale for multimodal and multigranularity perception in determining which parts contribute most to video classification.

Knowledge Distillation

Since SCAM+ requires video tags, it cannot be applied directly to unlabeled "in-the-wild" videos. To address this, the pseudofixations generated by SCAM+ serve as training targets for a "student" network, STANet+. Once trained, STANet+ can predict fixations on any video using only visual-audio input, without needing semantic tags.

Experimental Results

The model was trained on the Audio-Visual Event (AVE) dataset and evaluated across six benchmarks (AVAD, Coutrot1, Coutrot2, DIEM, SumMe, and ETMD). Key findings include:

  • Competitive Performance: On the AVAD dataset, STANet+ achieved a Correlation Coefficient (CC) of 0.490, rivaling fully supervised models like SalGAN (0.491) despite using no real fixation data during training.
  • Audio Integration: An "Audio Switch" module effectively filters irrelevant background noise, ensuring the model only attends to audio cues that are semantically relevant to the visual scene.
  • Generalization: The student network demonstrated robust performance on unseen videos, proving the effectiveness of the knowledge distillation process.

Conclusion and Future Work

This work demonstrates that weakly supervised learning using video tags is a viable alternative to expensive eye-tracking data for VFP. While the current model assumes a single primary tag per video, future research will explore multi-tag supervision and more complex decoder architectures to further bridge the gap with fully supervised systems. The ability to generate high-quality fixations from unlimited, low-cost data sources represents a significant step forward for practical visual-auditory saliency detection.

Summary updated

Abstract

Thanks to the rapid advances in deep learning techniques and the wide availability of large-scale training sets, the performance of video saliency detection models has been improving steadily and significantly. However, deep learning-based visualaudio fixation prediction is still in its infancy. At present, only a few visual-audio sequences have been furnished, with real fixations being recorded in real visual-audio environments. Hence, it would be neither efficient nor necessary to recollect real fixations under the same visual-audio circumstances. To address this problem, this paper promotes a novel approach in a weakly supervised manner to alleviate the demand of large-scale training sets for visual-audio model training. By using only the video category tags, we propose the selective class activation mapping (SCAM) and its upgrade (SCAM+). In the spatial-temporal-audio circumstance, the former follows a coarse-to-fine strategy to select the most discriminative regions, and these regions are usually capable of exhibiting high consistency with the real human-eye fixations. The latter equips the SCAM with an additional multi-granularity perception mechanism, making the whole process more consistent with that of the real human visual system. Moreover, we distill knowledge from these regions to obtain complete new spatial-temporal-audio (STA) fixation prediction (FP) networks, enabling broad applications in cases where video tags are not available. Without resorting to any real human-eye fixation, the performances of these STA FP networks are comparable to those of fully supervised networks. The code and results are publicly available at https://github.com/guotaowang/STANet.
BibTeX
@misc{wang2021weakly,
  author = {Guotao Wang and Chenglizhao Chen and Deng-Ping Fan and Aimin Hao and Hong Qin},
  title = {{Weakly Supervised Visual-Auditory Saliency Detection with Multigranularity Perception}},
  year = {2021},
  eprint = {2112.13697},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.13697},
}
Figure