Spherical Vision Transformers for Audio-Visual Saliency Prediction in 360$^{\circ }$∘ Videos

Mert Cokelek, Halit Ozsoy, Nevrez Imamoglu, Cagri Ozcinar, Inci Ayhan, Erkut Erdem, Aykut ErdemIEEE TPAMI 2026 · DOI

TL;DR

This paper introduces SalViT360-AV, a novel framework that leverages Spherical Vision Transformers to predict where people look in 360° videos, crucially incorporating spatial audio cues. This advancement is critical for optimizing immersive experiences in VR and other applications by better understanding human attention in complex, spherical visual environments.

What Problem Does This Paper Address?

The rise of 360° videos and VR has created a need to understand viewer attention in these immersive environments. Predicting visual saliency (where people look) is key for tasks like video compression and adaptive streaming. However, 360° videos present unique challenges: spherical distortion in standard projections (like equirectangular) and the significant, yet often overlooked, role of spatial audio in guiding attention. Existing models either struggle with spherical geometry or fail to integrate audio effectively. This paper aims to bridge this gap by developing models that can accurately predict saliency in 360° videos, explicitly considering both visual and spatial audio information.

Methods

The core of the proposed solution is SalViT360-AV, an extension of their previous visual-only model, SalViT360.

  • Spherical Vision Transformers (SalViT360): This model uses a Vision Transformer (ViT) architecture adapted for 360° videos. It addresses spherical distortion by projecting video frames into multiple "tangent images" using gnomonic projection. These tangent images are then processed by a transformer with "spherical geometry-aware spatio-temporal attention" layers to capture both local visual details and global context across the spherical scene.
  • Audio Integration (SalViT360-AV): To incorporate spatial audio, SalViT360-AV uses a two-stream approach. The audio stream processes first-order ambisonic (4-channel) audio. Crucially, the ambisonic audio is rotated to align with each tangent viewport, creating "view-aligned" mono audio signals. These signals are then fed through a pre-trained audio backbone (like PaSST) and their features are fused with the visual features using "transformer adapters" (inspired by AdaptFormer). This allows for parameter-efficient fine-tuning and effective cross-modal fusion.

Datasets

  • YT360-EyeTracking (New Dataset): The authors introduce a new, large-scale dataset specifically for 360° audio-visual saliency prediction. It contains 81 videos annotated with eye-tracking data under three audio conditions: mute, mono, and ambisonic. This dataset is designed to systematically study the impact of different audio modalities.
  • Benchmark Datasets: The models were also evaluated on existing datasets, including:
  • VR-EyeTracking (used for visual-only training and evaluation)
    • PVS-HMEM
    • 360AV-HM
    • AVS-ODV
    • SVGC-AVA

Key Insights and Ideas

  • Spatial Audio is Crucial: The most significant insight is the profound impact of spatial audio on guiding viewer attention. The study demonstrates that models incorporating ambisonic audio significantly outperform visual-only or 2D audio-visual models, especially in complex scenes where audio cues are directional.
  • View-Aligned Audio Processing: Simply feeding raw ambisonic audio isn't as effective as rotating it to align with the viewer's current perspective (tangent viewport). This "view-aligned" processing allows for more direct and meaningful audio-visual fusion.
  • Transformer Adapters for Efficient Fusion: Using parameter-efficient transformer adapters for audio integration is a key methodological contribution, allowing for effective multi-modal learning without retraining the entire large visual model.
  • Spherical Geometry Matters: Standard projection methods for 360° videos introduce distortions. The use of gnomonic projection to create tangent images and spherical geometry-aware attention mechanisms in SalViT360 effectively handles these distortions.

Key Results

  • State-of-the-Art Performance: SalViT360-AV achieves state-of-the-art results on multiple benchmark datasets, including the newly introduced YT360-EyeTracking dataset, for audio-visual saliency prediction in 360° videos.
  • Significant Improvement with Spatial Audio: On the YT360-EyeTracking dataset, SalViT360-AV consistently outperformed visual-only and 2D audio-visual baselines across all metrics (NSS, KLD, CC, SIM). For example, it achieved an NSS score of 2.449 compared to 1.738 for SalViT360 (visual-only) on this dataset.
  • Cross-Dataset Generalization: The model demonstrates strong generalization capabilities, performing well on unseen datasets with varying audio complexities and content types.
  • Inter-Subject Consistency: Higher consistency in viewer fixations was observed under ambisonic audio conditions, underscoring its role in synchronizing attention.

Limitations and Future Work

  • Computational Cost: While the adapter approach is parameter-efficient, the overall transformer-based architecture can still be computationally intensive.
  • Audio Backbone Choice: While PaSST was chosen for its availability, other audio backbones might offer further improvements. The paper notes that AVSA, specifically trained for FOA, showed promising results.
  • Future Work: The authors suggest that their models have broader applications beyond saliency prediction, such as improving audio-visual quality assessment, generating immersive navigation paths, and guiding compression for perceptually important regions. Further exploration of different audio encoding techniques and more complex audio-visual interactions could also be fruitful.

Summary updated

BibTeX
@article{cokelek2026spherical,
  author = {Mert Cokelek and Halit Ozsoy and Nevrez Imamoglu and Cagri Ozcinar and Inci Ayhan and Erkut Erdem and Aykut Erdem},
  title = {{Spherical Vision Transformers for Audio-Visual Saliency Prediction in 360$^\{\textbackslash\{\}circ \}$∘ Videos}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2026},
  doi = {10.1109/tpami.2025.3604091},
}
Figure