Abstract
Predicting human eye fixations in videos with audio, known as Video Fixation Prediction (VFP), typically requires expensive and labor-intensive eye-tracking data. This paper introduces a novel weakly supervised framework that leverages simple video category tags (e.g., "horse," "violin") to generate high-quality "pseudofixations." By employing Selective Class Activation Mapping (SCAM) and a multigranularity perception module (SCAM+), the framework achieves performance comparable to state-of-the-art fully supervised models without the need for real human-eye fixation data during training.
Introduction
Video Fixation Prediction (VFP) aims to predict the scattered coordinates of human gaze patterns, a task significantly more complex than standard Video Salient Object Detection (VSOD), which focuses on segmenting entire objects with clear boundaries. The primary challenge in VFP is the scarcity of large-scale datasets containing real human fixations, especially in visual-audio environments, due to the high cost of specialized eye-tracking hardware and human subjects.
This research bridges the gap by using easily obtainable video category tags as a proxy for supervision. The core hypothesis is that the regions a machine uses to classify a video often align with the regions that attract human attention.
Methodology
Overview of the Framework
The proposed approach moves from simple semantic labels to complex fixation predictions through a multi-stage pipeline involving pseudofixation generation and knowledge distillation.
Selective Class Activation Mapping (SCAM)
SCAM identifies discriminative regions by decoupling video data into three streams: Spatial (S), Spatiotemporal (ST), and Spatial-Audio (SA). These streams are selectively fused based on classification confidence to generate initial fixation maps.
SCAM+ and Multigranularity Perception
To mimic the global nature of human attention, the authors developed SCAM+, which incorporates multigranularity perception. This module uses a Conservative Graph Convolutional Network (C-GCN) to reason across different temporal and semantic scales:
- Short-term memory: Neighboring frames.
- Long-term memory: Distant frames within the same video.
- Cross-term memory: Frames from different videos sharing the same category tag.
Knowledge Distillation
Since SCAM+ requires video tags, it cannot be applied directly to unlabeled "in-the-wild" videos. To address this, the pseudofixations generated by SCAM+ serve as training targets for a "student" network, STANet+. Once trained, STANet+ can predict fixations on any video using only visual-audio input, without needing semantic tags.
Experimental Results
The model was trained on the Audio-Visual Event (AVE) dataset and evaluated across six benchmarks (AVAD, Coutrot1, Coutrot2, DIEM, SumMe, and ETMD). Key findings include:
- Competitive Performance: On the AVAD dataset, STANet+ achieved a Correlation Coefficient (CC) of 0.490, rivaling fully supervised models like SalGAN (0.491) despite using no real fixation data during training.
- Audio Integration: An "Audio Switch" module effectively filters irrelevant background noise, ensuring the model only attends to audio cues that are semantically relevant to the visual scene.
- Generalization: The student network demonstrated robust performance on unseen videos, proving the effectiveness of the knowledge distillation process.
Conclusion and Future Work
This work demonstrates that weakly supervised learning using video tags is a viable alternative to expensive eye-tracking data for VFP. While the current model assumes a single primary tag per video, future research will explore multi-tag supervision and more complex decoder architectures to further bridge the gap with fully supervised systems. The ability to generate high-quality fixations from unlimited, low-cost data sources represents a significant step forward for practical visual-auditory saliency detection.
Summary updated