TL;DR
This paper introduces a novel audiovisual saliency model that integrates audio cues with visual information to predict human attention in complex, uncategorized video sequences. By correlating audio features with visual motion, the model demonstrates improved performance over purely visual saliency models, highlighting the importance of sound in guiding our gaze.
What Problem Does This Paper Address?
Traditional saliency models, which aim to predict where humans look in visual scenes, often treat videos as sequences of static images or focus on specific video categories. This approach overlooks the significant role audio plays in guiding human attention, especially in dynamic and complex environments. Existing models that do incorporate audio are often limited to specific types of videos (e.g., movies, conversations), failing to generalize to a wider range of content. This paper addresses the need for a generic audiovisual saliency model that can effectively predict attention in diverse, uncategorized video sequences by leveraging the interplay between sound and visual motion.
Methods
The proposed audiovisual saliency model is built upon a framework that combines visual, audio, and motion information. The core idea is to create three distinct saliency maps—visual, audio, and motion—and then fuse them into a unified audiovisual saliency map.
- Feature Extraction:
- Audio: Audio energy descriptors are extracted by analyzing the spectrogram of the audio signal, synchronized with video frames.
- Visual: Motion descriptors are derived from optical flow estimation. Specifically, they compute forward and backward optical flows to capture pixel motion.
- Saliency Map Computation:
- Visual Saliency: A classical visual saliency model based on Markov chains is employed, considering features like intensity, color, orientation, flicker, and motion.
- Audio Saliency: This is a key novelty. An audio-video correlation method is used, where audio and motion descriptors are transformed into a common rank correlation space. Hamming distance is then used to relate audio signals to objects, indicating audio saliency.
- Motion Map: This map is generated from the mean optical flow per frame, with adaptive thresholding to filter out inconsequential low motion.
- Normalization and Combination: The three computed maps are normalized to a [0, 1] range and then linearly combined to produce the final audiovisual saliency map.
Datasets
The primary dataset used for evaluation is the Dynamic Images and Eye Movements (DIEM) dataset. This dataset contains 85 videos with or without audio, collected with eye-tracking experiments from 250 participants. For their evaluation, the authors randomly selected 25 videos with audio from this dataset. No new benchmark datasets were introduced.
Key Insights and Ideas
- Audio is Crucial for Saliency in Complex Videos: The paper strongly supports the hypothesis that incorporating audio information significantly improves saliency prediction in uncategorized video sequences, outperforming purely visual models.
- Audio-Visual Correlation as a Saliency Indicator: The novel approach of correlating audio features with visual motion to derive an audio saliency map is a key contribution, suggesting that where sound originates visually is a strong predictor of attention.
- Generic Model for Uncategorized Videos: The model's ability to work across various video genres, rather than being limited to specific categories like conversations or movies, is a significant step towards more robust and generalizable saliency prediction.
- Motion as a Bridge: Using motion descriptors as a link between audio and visual information is an effective strategy for localizing sound sources within the visual scene.
Key Results
The proposed audiovisual saliency model was evaluated against two state-of-the-art visual saliency models using metrics like AUC, DKL, NSS, and CC.
- Performance Improvement: The proposed model achieved a higher average AUC score (0.739) compared to the visual models (0.716 and 0.712).
- Lower Dissimilarity: The model also resulted in a lower DKL score (4.153) compared to the visual models (4.255 and 6.437), indicating a better approximation of ground truth fixations.
- Comparable Performance on Other Metrics: While the CC and NSS scores were slightly lower than one of the comparison methods, the overall results suggest that the audiovisual model makes better eye movement predictions.
- Qualitative Examples: Figure 2 shows qualitative comparisons where the audiovisual model correctly identified a talking face as salient in a video where visual models failed to do so, demonstrating the impact of audio cues.
Limitations and Future Work
- Dataset Subset: The evaluation was performed on a subset of 25 videos from the DIEM dataset. The authors suggest evaluating on the entire dataset and other available datasets to further validate their findings.
- Computational Complexity: The model exhibits higher time complexity, partly due to the dense optical flow computation.
- Object Segmentation and Tracking Challenges: The difficulty in accurately segmenting and tracking multiple interacting objects, especially with factors like motion blur and occlusion, affects the performance of the audio saliency map.
- Multiple/Off-Screen Audio Sources: Locating audio sources becomes more challenging with multiple or off-screen sounds, impacting the model's performance.
- Future Directions:
- Improving segmentation and tracking techniques for better audio saliency maps.
- Exploring more sophisticated visual saliency models.
- Investigating more suitable combination techniques for fusing the saliency maps.
- Incorporating more advanced audio features.
Summary updated