AVSegFormer: Audio-Visual Segmentation with Transformer

Shengyi Gao, Zhe Chen, Guo Chen, Wenhai Wang, Tong Lu

Abstract

The combination of audio and vision has long been a topic of interest in the multi-modal community. Recently, a new audio-visual segmentation (AVS) task has been introduced, aiming to locate and segment the sounding objects in a given video. This task demands audio-driven pixel-level scene understanding for the first time, posing significant challenges. In this paper, we propose AVSegFormer, a novel framework for AVS tasks that leverages the transformer architecture. Specifically, we introduce audio queries and learnable queries into the transformer decoder, enabling the network to selectively attend to interested visual features. Besides, we present an audio-visual mixer, which can dynamically adjust visual features by amplifying relevant and suppressing irrelevant spatial channels. Additionally, we devise an intermediate mask loss to enhance the supervision of the decoder, encouraging the network to produce more accurate intermediate predictions. Extensive experiments demonstrate that AVSegFormer achieves state-of-the-art results on the AVS benchmark.

Introduction: The Challenge of Pixel-Level Audio-Visual Understanding

While humans can effortlessly identify which car is honking or which person is speaking in a crowded scene, AI has traditionally struggled with fine-grained multi-modal understanding. Previous audio-visual tasks often focused on coarse classification (e.g., "is a dog barking?"). Audio-Visual Segmentation (AVS) elevates this challenge by requiring the model to delineate the precise pixel-level shape of sounding objects.

Existing methods often rely on simple fusion techniques, such as concatenating audio features with image features. However, these models frequently fail in complex environments where multiple objects coexist or audio signals are dense, struggling to "untangle" which sound belongs to which visual entity.

Overview of the three AVS tasks: Single Sound Source (S4), Multiple Sound Source (MS3), and Audio-Visual Semantic Segmentation (AVSS).

Methodology: The AVSegFormer Framework

AVSegFormer departs from traditional convolutional fusion in favor of a transformer-centric architecture. The framework is built upon four primary pillars:

  1. Transformer Encoder: Generates initial mask features from video frames.
  2. Dense Audio-Visual Mixer: Acts as the attention hub. It utilizes Channel-Attention (CHA) to selectively amplify visual features correlating with the audio while suppressing irrelevant background noise.
  3. Query Generator: Conditioned on the audio signal, this component generates a set of queries to "hunt" for specific sounding objects.
  4. Sparse Audio-Visual Decoder: Matches these queries against visual features to separate different sound sources, such as distinguishing a guitar's sound from a singer's voice.

A key innovation is the Auxiliary Mixing Loss ($L_{mix}$), which supervises the mixer to ensure it correctly highlights relevant regions before the final segmentation, serving as a critical sanity check for multi-modal fusion.

Key Insights

  • Bidirectional Conditioning: Unlike models that merely "inject" audio into vision, AVSegFormer uses audio to adjust visual features and then uses those features to refine audio-based queries.
  • Channel Attention: The researchers found that weighting visual channels based on audio is more effective than standard spatial cross-attention, allowing the model to re-calibrate its visual filters.
  • Learnable Queries: Using a fixed number of learnable queries (optimized at 300) allows the model to adapt to varying numbers of sounding objects without architectural changes.

Experiments and Results

The model was evaluated on the AVSBench suite, covering three distinct scenarios: Single Sound Source (S4), Multiple Sound Source (MS3), and Audio-Visual Semantic Segmentation (AVSS).

Quantitative Performance

AVSegFormer established new state-of-the-art records on AVSBench, significantly outperforming previous baselines:

  • S4: Achieved 76.45 mIoU, a 3.66-point improvement over the previous state-of-the-art.
  • MS3: Reached 61.33 mIoU, a massive jump from the baseline's 54.00.
  • AVSS: Achieved 37.31 mIoU, outperforming the baseline by 7.54 points.

Qualitative Analysis

Qualitatively, the model excels at ignoring "distractor" objects—such as a silent instrument—while precisely capturing the boundaries of active sound sources. It demonstrates superior robustness in complex scenes where multiple objects are present but only a subset is making noise.

Qualitative result of Audio-Visual Semantic Segmentation (AVSS) showing precise pixel-level identification of both the singer and the guitar.

Conclusion

AVSegFormer demonstrates a significant leap in pixel-perfect audio-visual understanding by leveraging a query-based transformer architecture and a dense audio-visual mixer. While the model is computationally intensive, its ability to untangle complex multi-modal scenes sets a new standard for the AVS task. Future work will focus on improving temporal consistency in rapidly changing scenes and refining the separation of overlapping signals in even more crowded environments.

Summary updated

Abstract

The combination of audio and vision has long been a topic of interest in the multi-modal community. Recently, a new audio-visual segmentation (AVS) task has been introduced, aiming to locate and segment the sounding objects in a given video. This task demands audio-driven pixel-level scene understanding for the first time, posing significant challenges. In this paper, we propose AVSegFormer, a novel framework for AVS tasks that leverages the transformer architecture. Specifically, we introduce audio queries and learnable queries into the transformer decoder, enabling the network to selectively attend to interested visual features. Besides, we present an audio-visual mixer, which can dynamically adjust visual features by amplifying relevant and suppressing irrelevant spatial channels. Additionally, we devise an intermediate mask loss to enhance the supervision of the decoder, encouraging the network to produce more accurate intermediate predictions. Extensive experiments demonstrate that AVSegFormer achieves state-of-the-art results on the AVS benchmark. The code is available at https://github.com/vvvb-github/AVSegFormer.
BibTeX
@inproceedings{gao2024avsegformer,
  author = {Shengyi Gao and Zhe Chen and Guo Chen and Wenhai Wang and Tong Lu},
  title = {{AVSegFormer: Audio-Visual Segmentation with Transformer}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=cPpPyvZmPo},
}
Figure