Segment beyond View: Handling Partially Missing Modality for Audio-Visual Semantic Segmentation

Renjie Wu, Hu Wang, Feras Dayoub, Hsiang-Ting ChenAAAI 2024 · DOI

Introduction

Current Augmented Reality (AR) and Head-Mounted Displays (HMDs) are often limited by "tunnel vision." While providing immersive overlays, their onboard cameras typically have a restricted Field of View (FoV), often narrower than human vision. This limitation poses a significant safety risk; for instance, an AI assistant cannot warn a pedestrian about a speeding car if it is behind them or outside the camera's frame.

The Problem: Partially Missing Modality

Most multimodal AI models assume that all sensors provide a complete view of the environment. This paper addresses the "partially missing modality" problem, where audio provides a 360-degree spatial context while video is restricted to a narrow slice. The goal is to use spatial audio cues to "fill in" the semantic map of areas the camera cannot see.

Overview of the SBV task and segmentation results.

Methods: The SBV Framework

The researchers proposed the Segment Beyond View (SBV) framework, which utilizes a distillation process called Omni2Ego. The architecture employs a Teacher-Student setup:

  1. The Teachers: During training, the model uses "privileged" information—a full 360-degree panoramic image and 8-channel professional audio—to establish ground truth for object locations.
  2. The Student: The real-world model receives only a limited FoV (simulating an AR headset) and 2-channel (binaural) audio.
  3. Knowledge Distillation: The student is trained to mimic the teachers, learning to associate specific acoustic patterns with the visual shapes and positions provided by the panoramic teacher.
Spatial audio mapping for the teacher model.

Key technical components include the Audio-Visual Feature Fusion Module (AVFFM), which aligns audio and visual features using attention mechanisms, and the Modality Reconstruction Loss (MRL). The MRL forces the model to attempt to reconstruct the missing panoramic view, encouraging the development of richer internal environmental representations.

Dataset and Evaluation

The study utilized the Omni Auditory Perception Dataset, containing 64,250 video clips of Zurich traffic recorded with 360-degree cameras and 8-channel audio. To handle the labor-intensive task of pixel-level labeling, the authors developed an automated "Sound-Making Objects Extraction" method using pre-trained models and background subtraction.

Key Results

The SBV model significantly outperformed existing state-of-the-art audio-visual segmentation models:

  • Overall Accuracy: SBV achieved a mean Intersection-over-Union (mIoU) of 53.0%, surpassing the previous best of 47.5%.
  • Out-of-View (OOV) Performance: In regions invisible to the camera, SBV reached 46.7% mIoU, a substantial improvement over the 39.0% baseline.
  • Qualitative Success: Visualizations demonstrate that SBV can predict the distinct shapes of vehicles located well outside the camera's FoV, whereas baseline models often fail to detect them or produce indistinct results.

Conclusion and Future Work

The SBV framework demonstrates that binaural audio contains sufficient spatial information to semantically segment objects in 2D space, provided the model is trained with panoramic visual guidance. Future work will focus on incorporating temporal information for trajectory tracking, optimizing the model for real-time performance on low-power hardware, and conducting user studies to evaluate real-world safety improvements.

Summary updated

BibTeX
@inproceedings{wu2024segment,
  author = {Renjie Wu and Hu Wang and Feras Dayoub and Hsiang-Ting Chen},
  title = {{Segment beyond View: Handling Partially Missing Modality for Audio-Visual Semantic Segmentation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i6.28426},
}
Figure