RA-SSU: Towards Fine-Grained Audio-Visual Learning with Region-Aware Sound Source Understanding

Muyi Sun, Yixuan Wang, Hong Wang, Chen Su, Man Zhang, Xingqun Qi, Qi Li, Zhenan Sun

TL;DR

The paper introduces RA-SSU (Region-Aware Sound Source Understanding), a new task that moves beyond coarse audio-visual matching to provide fine-grained, frame-level segmentation masks and textual descriptions for sounding objects. To solve this, the authors developed SSUFormer, a multi-modal Transformer that achieves state-of-the-art performance by explicitly linking spatial masks with semantic captions.

What Problem Does This Paper Address?

Traditional Audio-Visual Learning (AVL) has focused on "coarse-grained" tasks. For example, Audio-Visual Correspondence (AVC) simply asks if a sound matches a video, and Sound Source Localization (SSL) often only provides a rough "heat map" of where a sound might be coming from.

The authors argue that these approaches fail in complex, dynamic real-world scenes where multiple objects might be making noise simultaneously. There is a lack of fine-grained understanding—specifically, the ability to precisely segment the sounding object (spatial) and describe its actions frame-by-frame (temporal/semantic). RA-SSU fills this gap by requiring a model to output both a pixel-level mask and a detailed description for every frame of a video.

Methods

The authors propose SSUFormer, a multi-modal input/output architecture. Its core components include:

  • Multi-Modal Encoders: It uses VGGish for audio features and PVT-v2 for visual features, which are then fused via cross-attention.
  • Mask Collaboration Module (MCM): This is a key innovation. Instead of treating segmentation and captioning as independent tasks, MCM uses the predicted masks as spatial guidance. By focusing the "attention" of the text generator on the masked regions, the model produces more accurate, region-specific descriptions.
  • Mixture of Hierarchical-prompted Experts (MoHE): To ensure captions remain consistent over time, this module integrates a Large Language Model (LLaVA) as a "linguistic expert." It uses a hierarchical router to balance long-term temporal cues from LLaVA with frame-wise visual features, preventing the "flickering" or inconsistent descriptions often seen in video captioning.
  • Regional Consistency Constraint (RCC): A specialized loss function that aligns the visual features of the masked region with the textual features of the description using a MaskCLIP-based encoder.

Datasets

Because no existing datasets provided both frame-level masks and descriptions, the authors created two new benchmarks:

  1. f-Music: 3,976 samples across 22 instrument-related scenes. It focuses on complex scenarios like musical ensembles where multiple instruments overlap.
  2. f-Lifescene: 6,156 samples across 61 categories of everyday life (e.g., cats meowing, lawnmowers running). This dataset is more challenging due to the diversity of objects and environments.

The annotation process was a "human-in-the-loop" pipeline: they used Segment Anything (SAM) and Track Anything (TAM) for masks, and Chat-Univi for initial text, followed by rigorous manual refinement.

Key Insights and Ideas

  • Masks as Semantic Anchors: The most significant insight is that spatial segmentation (the "where") is a prerequisite for high-quality semantic description (the "what"). By forcing the model to "look" at the mask while "writing" the caption, the authors reduced hallucinations.
  • Task-Specific vs. General Purpose: While Multi-modal Large Language Models (MLLMs) like GPT-4V are powerful, they often struggle with precise spatial grounding in video. SSUFormer demonstrates that a specialized, task-aware architecture can significantly outperform general-purpose MLLMs on fine-grained benchmarks.
  • Co-Consistency: The authors found a "virtuous cycle" during training: better captioning results actually improved the segmentation masks. This suggests that semantic understanding helps the model better define the physical boundaries of a sounding object.

Key Results

SSUFormer established a new state-of-the-art (SOTA) on both datasets:

  • Segmentation: On f-Music, it achieved an mIoU of 0.6280, significantly higher than previous SSL models like AVIS (0.5968).
  • Captioning: It crushed general-purpose MLLMs in description accuracy. On the combined benchmark, SSUFormer reached a BLEU score of 0.4676, nearly doubling the performance of PG-Video-LLaVA (0.2300).
  • Ablation Studies: Removing the MCM or MoHE modules led to a noticeable drop in performance, proving that the interaction between the mask and the text is critical for the RA-SSU task.

Limitations and Future Work

The authors acknowledge several hurdles:

  • Open-Set Perception: The model is currently limited to the categories present in the training data. It cannot yet describe or segment "unseen" sounding objects with the same level of precision.
  • Audio Sensitivity: The system can be sensitive to background noise. If the audio environment changes drastically or becomes too "messy," the segmentation accuracy drops.
  • Data Complexity: Creating these fine-grained datasets is incredibly labor-intensive. Future work will likely look into semi-supervised or self-supervised ways to scale up the data without requiring thousands of hours of manual refinement.

Summary updated

Abstract

Audio-Visual Learning (AVL) is one fundamental task of multi-modality learning and embodied intelligence, displaying the vital role in scene understanding and interaction. However, previous researchers mostly focus on exploring downstream tasks from a coarse-grained perspective (e.g., audio-visual correspondence, sound source localization, and audio-visual event localization). Considering providing more specific scene perception details, we newly define a fine-grained Audio-Visual Learning task, termed Region-Aware Sound Source Understanding (RA-SSU), which aims to achieve region-aware, frame-level, and high-quality sound source understanding. To support this goal, we innovatively construct two corresponding datasets, i.e. fine-grained Music (f-Music) and fine-grained Lifescene (f-Lifescene), each containing annotated sound source masks and frame-by-frame textual descriptions. The f-Music dataset includes 3,976 samples across 22 scene types related to specific application scenarios, focusing on music scenes with complex instrument mixing. The f-Lifescene dataset contains 6,156 samples across 61 types representing diverse sounding objects in life scenarios. Moreover, we propose SSUFormer, a Sound-Source Understanding TransFormer benchmark that facilitates both the sound source segmentation and sound region description with a multi-modal input and multi-modal output architecture. Specifically, we design two modules for this framework, Mask Collaboration Module (MCM) and Mixture of Hierarchical-prompted Experts (MoHE), to respectively enhance the accuracy and enrich the elaboration of the sound source description. Extensive experiments are conducted on our two datasets to verify the feasibility of the task, evaluate the availability of the datasets, and demonstrate the superiority of the SSUFormer, which achieves SOTA performance on the Sound Source Understanding benchmark.
BibTeX
@misc{sun2026ra,
  author = {Muyi Sun and Yixuan Wang and Hong Wang and Chen Su and Man Zhang and Xingqun Qi and Qi Li and Zhenan Sun},
  title = {{RA-SSU: Towards Fine-Grained Audio-Visual Learning with Region-Aware Sound Source Understanding}},
  year = {2026},
  eprint = {2603.09809},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.09809},
}
Figure