Complementary and Contrastive Learning for Audio-Visual Segmentation

Sitong Gong, Yunzhi Zhuge, Lu Zhang, Pingping Zhang, Huchuan Lu

Introduction

Audio-Visual Segmentation (AVS) is an emerging computer vision task that requires a model to identify and generate pixel-level masks for objects in a video that are actively producing sound. Unlike standard video object segmentation, AVS is inherently cross-modal; it requires the system to understand the semantic link between an auditory signal (like a bark or a piano chord) and a visual entity. This task is critical for applications ranging from video editing to autonomous robotics and assistive technologies.

Problem Statement

Current approaches to AVS, often based on Convolutional Neural Networks (CNNs) or basic Transformers, face two primary hurdles. First, they typically employ unidirectional or late-stage fusion, where audio information is simply injected into visual features. This often leads to a "lazy" model that relies too heavily on one modality, failing to achieve true cross-modal alignment. Second, many models process video frames in isolation. By ignoring the temporal flow of a scene, these models struggle to maintain consistency when objects are partially occluded or when the sound source moves or stops.

Comparison of AVS task overview and different methodological approaches, highlighting the proposed CCFormer's bidirectional and temporal integration.

Methodology: The CCFormer Framework

To address these limitations, the authors introduce CCFormer (Complementary and Contrastive Transformer). This framework is designed to capture both local spatial details and global temporal coherence through three integrated components.

1. Early Integration Module (EIM)

Unlike previous "late-fusion" models, CCFormer utilizes the EIM immediately following initial feature extraction. The EIM employs a parallel bilateral architecture that facilitates bidirectional communication: visual features are refined using audio context, and audio features are simultaneously updated using visual cues. This early-stage filtering helps the model discard irrelevant visual background noise and focus the audio queries on the specific objects responsible for the sound.

2. Multi-query Transformer Module (MTM)

The MTM serves as the spatio-temporal reasoning engine. Instead of using static or random queries, it employs an Attention Query Generator to produce dynamic intra-frame queries based on the audio signal. The decoder then alternates between two critical interaction layers:

  • Intra-frame Cross-modal Interaction (ICI): This layer focuses on spatial localization, pinpointing the sounding object within a single frame.
  • Inter-frame Temporal Interaction (ITI): This layer tracks the object across the entire video sequence, ensuring that the generated masks remain temporally consistent even during complex scene transitions.

3. Bi-modal Contrastive Learning (BCL)

To further tighten the alignment between modalities, CCFormer introduces a frame-level contrastive loss. The BCL pulls together the original audio features and the visually-enhanced audio features from the same frame while pushing apart features from different samples. This ensures that the model operates within a unified, highly discriminative feature space.

Detailed architecture of CCFormer, showing the flow from visual/audio encoders through the Early Integration Module and the Multi-query Transformer Decoder.

Experiments and Results

The model was rigorously tested on the AVSBench benchmarks, including the AVSBench-object (S4 and MS3 settings) and the more complex AVSBench-semantic dataset.

  • MS3 (Multiple Sound Source Segmentation): In this challenging scenario involving multiple objects and transitions, CCFormer achieved a significant performance leap. Using a ResNet-50 backbone, it improved the Jaccard index ($\mathcal{M_J}$) by 9.0% and the F-score ($\mathcal{M_F}$) by 7.4% over previous state-of-the-art methods.
  • AVSBench-semantic: On the task of simultaneous localization and classification across 70 categories, CCFormer improved $\mathcal{M_J}$ by 5.4% and $\mathcal{M_F}$ by 6.6%.

Qualitative analysis reveals that CCFormer is particularly adept at ignoring "silent" objects—such as a person holding an instrument without playing it—and maintaining mask stability during rapid scene changes, where earlier methods often produced "hallucinated" or flickering masks.

Conclusion and Future Outlook

CCFormer demonstrates that bidirectional early fusion and explicit temporal modeling are essential for robust Audio-Visual Segmentation. By treating audio and video as complementary signals that must be aligned at both the spatial and temporal levels, the framework sets a new standard for the field. While the model currently faces challenges with completely silent frames or highly obscured sources, future research aims to expand this architecture to include natural language, potentially allowing for complex, instruction-based video segmentation and reasoning.

Summary updated

BibTeX
@article{gong2025complementary,
  author = {Sitong Gong and Yunzhi Zhuge and Lu Zhang and Pingping Zhang and Huchuan Lu},
  title = {{Complementary and Contrastive Learning for Audio-Visual Segmentation}},
  journal = {IEEE Trans. Multimedia},
  year = {2025},
  doi = {10.1109/tmm.2025.3599048},
}

Loading related content...

Figure