TL;DR
The paper introduces RA-SSU (Region-Aware Sound Source Understanding), a new task that moves beyond coarse audio-visual matching to provide fine-grained, frame-level segmentation masks and textual descriptions for sounding objects. To solve this, the authors developed SSUFormer, a multi-modal Transformer that achieves state-of-the-art performance by explicitly linking spatial masks with semantic captions.
What Problem Does This Paper Address?
Traditional Audio-Visual Learning (AVL) has focused on "coarse-grained" tasks. For example, Audio-Visual Correspondence (AVC) simply asks if a sound matches a video, and Sound Source Localization (SSL) often only provides a rough "heat map" of where a sound might be coming from.
The authors argue that these approaches fail in complex, dynamic real-world scenes where multiple objects might be making noise simultaneously. There is a lack of fine-grained understanding—specifically, the ability to precisely segment the sounding object (spatial) and describe its actions frame-by-frame (temporal/semantic). RA-SSU fills this gap by requiring a model to output both a pixel-level mask and a detailed description for every frame of a video.
Methods
The authors propose SSUFormer, a multi-modal input/output architecture. Its core components include:
- Multi-Modal Encoders: It uses VGGish for audio features and PVT-v2 for visual features, which are then fused via cross-attention.
- Mask Collaboration Module (MCM): This is a key innovation. Instead of treating segmentation and captioning as independent tasks, MCM uses the predicted masks as spatial guidance. By focusing the "attention" of the text generator on the masked regions, the model produces more accurate, region-specific descriptions.
- Mixture of Hierarchical-prompted Experts (MoHE): To ensure captions remain consistent over time, this module integrates a Large Language Model (LLaVA) as a "linguistic expert." It uses a hierarchical router to balance long-term temporal cues from LLaVA with frame-wise visual features, preventing the "flickering" or inconsistent descriptions often seen in video captioning.
- Regional Consistency Constraint (RCC): A specialized loss function that aligns the visual features of the masked region with the textual features of the description using a MaskCLIP-based encoder.
Datasets
Because no existing datasets provided both frame-level masks and descriptions, the authors created two new benchmarks:
- f-Music: 3,976 samples across 22 instrument-related scenes. It focuses on complex scenarios like musical ensembles where multiple instruments overlap.
- f-Lifescene: 6,156 samples across 61 categories of everyday life (e.g., cats meowing, lawnmowers running). This dataset is more challenging due to the diversity of objects and environments.
The annotation process was a "human-in-the-loop" pipeline: they used Segment Anything (SAM) and Track Anything (TAM) for masks, and Chat-Univi for initial text, followed by rigorous manual refinement.
Key Insights and Ideas
- Masks as Semantic Anchors: The most significant insight is that spatial segmentation (the "where") is a prerequisite for high-quality semantic description (the "what"). By forcing the model to "look" at the mask while "writing" the caption, the authors reduced hallucinations.
- Task-Specific vs. General Purpose: While Multi-modal Large Language Models (MLLMs) like GPT-4V are powerful, they often struggle with precise spatial grounding in video. SSUFormer demonstrates that a specialized, task-aware architecture can significantly outperform general-purpose MLLMs on fine-grained benchmarks.
- Co-Consistency: The authors found a "virtuous cycle" during training: better captioning results actually improved the segmentation masks. This suggests that semantic understanding helps the model better define the physical boundaries of a sounding object.
Key Results
SSUFormer established a new state-of-the-art (SOTA) on both datasets:
- Segmentation: On f-Music, it achieved an mIoU of 0.6280, significantly higher than previous SSL models like AVIS (0.5968).
- Captioning: It crushed general-purpose MLLMs in description accuracy. On the combined benchmark, SSUFormer reached a BLEU score of 0.4676, nearly doubling the performance of PG-Video-LLaVA (0.2300).
- Ablation Studies: Removing the MCM or MoHE modules led to a noticeable drop in performance, proving that the interaction between the mask and the text is critical for the RA-SSU task.
Limitations and Future Work
The authors acknowledge several hurdles:
- Open-Set Perception: The model is currently limited to the categories present in the training data. It cannot yet describe or segment "unseen" sounding objects with the same level of precision.
- Audio Sensitivity: The system can be sensitive to background noise. If the audio environment changes drastically or becomes too "messy," the segmentation accuracy drops.
- Data Complexity: Creating these fine-grained datasets is incredibly labor-intensive. Future work will likely look into semi-supervised or self-supervised ways to scale up the data without requiring thousands of hours of manual refinement.
Summary updated