AVSegFormer: Audio-Visual Segmentation with Transformer

Shengyi Gao, Zhe Chen, Guo Chen, Wenhai Wang, Tong Lu

TL;DR

This paper introduces AVSegFormer, a novel transformer-based framework for audio-visual segmentation (AVS) that excels at understanding the intricate relationship between sound and visuals. By employing a dense audio-visual mixer and a sparse audio-visual decoder, AVSegFormer achieves state-of-the-art performance on challenging AVS benchmarks, particularly in scenarios with multiple sound sources.

What Problem Does This Paper Address?

Audio-visual segmentation (AVS) aims to precisely identify and outline objects in a video that are producing sound, based on the accompanying audio. This task requires a deep understanding of how audio cues relate to visual elements at a pixel level. Existing methods struggle with dynamic scenarios, such as evolving sounds, multiple objects producing sound simultaneously, and the fine-grained correlation between audio and visual information. These challenges make it difficult to accurately segment objects, especially when multiple sound sources are present, leading to imprecise localization and segmentation.

Methods

AVSegFormer leverages a transformer architecture to address the limitations of previous AVS methods. Its core components include:

  • Transformer Encoder: This module builds an initial "mask feature" by processing multi-scale visual features from a backbone network.
  • Dense Audio-Visual Mixer: This novel component dynamically adjusts visual features based on audio cues. It uses a channel-attention mechanism to selectively amplify relevant visual channels and suppress irrelevant ones, guided by the audio. An auxiliary "mixing loss" is introduced to supervise this mixer, encouraging it to focus on useful information within complex audio semantics.
  • Sparse Audio-Visual Decoder: This component generates sparse, audio-conditioned queries. These queries are then used to implicitly separate audio sources and match them with optimal visual features through a cross-modal transformer decoder. This approach allows for disentangling multiple sound sources.
  • Bidirectional Conditional Fusion: The framework combines the outputs of the dense mixer and the sparse decoder to create a robust, bidirectional conditional multi-modal representation, leading to improved segmentation.

Datasets

The paper evaluates AVSegFormer on two primary datasets:

  • AVSBench-Object (Zhou et al., 2022): This dataset is designed for audio-visual segmentation and includes pixel-level annotations. It has two subsets:
  • S4 (Single Sound Source Segmentation): For scenarios with one sound source.
  • MS3 (Multiple Sound Source Segmentation): For scenarios with multiple sound sources.
  • AVSBench-Semantic (Zhou et al., 2023): An extension of AVSBench-Object, this dataset provides additional semantic labels for audio-visual semantic segmentation (AVSS) and features longer video clips.

Key Insights and Ideas

  • The power of transformers for audio-visual fusion: The paper demonstrates that transformers are highly effective for capturing the complex, cross-modal relationships required for AVS.
  • Dense-sparse complementarity: Combining a dense audio-visual mixer (for fine-grained visual feature enhancement) with a sparse audio-visual decoder (for disentangling sound sources) leads to a more comprehensive and robust representation.
  • Channel attention for audio-guided visual enhancement: The channel-attention mixer proves crucial for adaptively highlighting relevant visual features based on audio cues, significantly improving performance over a cross-attention mixer.
  • Auxiliary mixing loss for better feature learning: The proposed auxiliary mixing loss effectively guides the audio-visual mixer, enhancing its ability to handle complex audio semantics and improve segmentation accuracy.
  • Learnable queries are beneficial for complex scenes: While learnable queries offer modest gains in single-sound-source scenarios, they provide significant improvements in multi-sound-source tasks, highlighting their importance for disentangling complex auditory environments.

Key Results

AVSegFormer achieves state-of-the-art results on the AVS benchmark across all three sub-tasks (S4, MS3, and AVSS).

  • S4 Subset: AVSegFormer-PVTv2 achieved an mIoU of 82.06 and an F-score of 89.9, outperforming the previous best method (LGVT) by 7.12 mIoU and 2.6 F-score.
  • MS3 Subset: AVSegFormer-PVTv2 surpassed the previous best method (iGAN) by 15.47 mIoU and 14.9 F-score.
  • AVSS Subset: AVSegFormer-PVTv2 achieved an mIoU of 36.66 and an F-score of 42.0, significantly outperforming AOT.
  • Qualitative Results: Visualizations show that AVSegFormer can more accurately segment sounding objects and outline their shapes compared to the AVSBench baseline, especially in complex scenes with multiple sound sources.

Limitations and Future Work

  • Number of Queries: The paper notes that performance generally increases with the number of queries, with 300 queries yielding optimal results. Further exploration into the optimal number and efficiency of queries could be a future direction.
  • Learnable Queries: While beneficial for MS3, the improvement from learnable queries in S4 was smaller. Investigating how to further leverage learnable queries for single-source scenarios might be interesting.
  • Dataset Size: The MS3 subset is noted as being quite small, which can pose challenges for training. Future work could involve exploring larger or more diverse multi-source datasets.
  • Generalization: While the paper demonstrates strong performance, further testing on a wider variety of real-world scenarios and datasets could reveal more about its generalization capabilities.

Summary updated

Abstract

Audio-visual segmentation (AVS) aims to locate and segment the sounding objects in a given video, which demands audio-driven pixel-level scene understanding. The existing methods cannot fully process the fine-grained correlations between audio and visual cues across various situations dynamically. They also face challenges in adapting to complex scenarios, such as evolving audio, the coexistence of multiple objects, and more. In this paper, we propose AVSegFormer, a novel framework for AVS that leverages the transformer architecture. Specifically, It comprises a dense audio-visual mixer, which can dynamically adjust interested visual features, and a sparse audio-visual decoder, which implicitly separates audio sources and automatically matches optimal visual features. Combining both components provides a more robust bidirectional conditional multi-modal representation, improving the segmentation performance in different scenarios. Extensive experiments demonstrate that AVSegFormer achieves state-of-the-art results on the AVS benchmark. The code is available at https://github.com/vvvb-github/AVSegFormer.
BibTeX
@inproceedings{gao2024avsegformer,
  author = {Shengyi Gao and Zhe Chen and Guo Chen and Wenhai Wang and Tong Lu},
  title = {{AVSegFormer: Audio-Visual Segmentation with Transformer}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=cPpPyvZmPo},
}
Figure