Abstract
Recently, significant progress has been made in multi-modal continual learning, aiming to learn new tasks sequentially in multi-modal settings while preserving performance on previously learned ones. However, existing methods mainly focus on coarse-grained tasks, with limitations in addressing modality entanglement in fine-grained continual learning settings. To bridge this gap, we introduce a novel Continual Audio-Visual Segmentation (CAVS) task, aiming to continuously segment new classes guided by audio. Through comprehensive analysis, two critical challenges are identified: 1) multi-modal semantic drift, where a sounding objects is labeled as background in sequential tasks; 2) co-occurrence confusion, where frequent co-occurring classes tend to be confused. In this work, a Collision-based Multi-modal Rehearsal (CMR) framework is designed to address these challenges. Specifically, for multi-modal semantic drift, a Multi-modal Sample Selection (MSS) strategy is proposed to select samples with high modal consistency for rehearsal. Meanwhile, for co-occurence confusion, a Collision-based Sample Rehearsal (CSR) mechanism is designed, allowing for the increase of rehearsal sample frequency of those confusable classes during training process. Moreover, we construct three audio-visual incremental scenarios to verify effectiveness of our method. Comprehensive experiments demonstrate that our method significantly outperforms single-modal continual learning methods.
Introduction
While Continual Learning (CL) has seen success in image classification and semantic segmentation, applying it to multi-modal settings like audio-visual segmentation introduces unique challenges. This paper introduces Continual Audio-Visual Segmentation (CAVS), a new task where models must learn to segment sounding objects in video sequences across a stream of tasks without forgetting old ones. The core difficulty lies in "modality entanglement," where the model struggles to maintain clear associations between audio and visual features as new data is introduced.
Challenges: Modality Entanglement
The researchers identify two primary hurdles that lead to catastrophic forgetting in multi-modal settings:
- Multi-modal Semantic Drift: In a sequential learning setup, an object learned in an earlier task (e.g., a "guitar") might be labeled as "background" in a later task. If the audio still contains the guitar sound, the model receives conflicting signals, causing it to overwrite previous knowledge.
- Co-occurrence Confusion: Certain objects frequently appear together, such as a woman playing a guitar. In fine-grained settings, the model may entangle the audio of the woman’s voice with the visual features of the guitar, leading to misclassification in subsequent tasks.
The CMR Framework
To address these challenges, the authors propose the Collision-based Multi-modal Rehearsal (CMR) framework. CMR moves beyond random sample rehearsal by implementing two specialized modules:
Multi-modal Sample Selection (MSS)
Instead of picking random samples for the memory buffer, MSS identifies samples with high modal consistency. It compares the performance of a visual-only model against an audio-visual model. If the audio significantly improves segmentation (or at least does not degrade it), the sample is deemed high-quality and saved for future rehearsal. This acts as a quality filter, ensuring the model's memory is populated with reliable associations.
Collision-based Sample Rehearsal (CSR)
A "collision" occurs when the old model’s prediction for a pixel disagrees with the new task's ground truth. By tracking these collisions, the system identifies which old classes are most "confusable" with new ones. CSR then dynamically increases the rehearsal frequency for those specific classes, forcing the model to prioritize disentangling these confused boundaries during training.
Experimental Evaluation
The researchers adapted the AVSBench dataset into three incremental scenarios: AVSBench-CI (Class Incremental), AVSBench-CIS (Single-object), and AVSBench-CIM (Multi-object). These were tested across both "Disjoint" and "Overlapped" settings.
Key Results
The CMR framework consistently outperformed state-of-the-art single-modal CL methods (like MiB and PLOP) and multi-modal methods (like EIR):
- Performance Stability: On the AVSBench-CI 60-10 split, CMR achieved a mean Intersection-over-Union (mIoU) of 27.6%, significantly higher than PLOP (20.1%) and MiB (18.2%).
- Fine-Grained Learning: In the difficult 65-1 split (learning classes one by one), CMR maintained an mIoU of 15.9%, while baseline methods often collapsed to near-zero performance.
- Qualitative Superiority: Visual results indicate that CMR produces finer, more complete segmentation masks, whereas baselines often "hallucinate" or miss objects entirely after several tasks.
Conclusion
By addressing the nuances of modality entanglement through intelligent sample selection and collision-based rehearsal, the CMR framework provides a robust solution for the CAVS task. While multi-target rehearsal in complex scenes (CIM) remains a challenge for future work, CMR sets a new benchmark for maintaining multi-modal associations in sequential learning environments.
Summary updated