SoundBeam: Target Sound Extraction Conditioned on Sound-Class Labels and Enrollment Clues for Increased Performance and Continuous Learning

Marc Delcroix, Jorge Bennasar Vázquez, Tsubasa Ochiai, Keisuke Kinoshita, Yasunori Ohishi, Shoko Araki

TL;DR

This paper introduces SoundBeam, a novel framework for Target Sound Extraction (TSE) that unifies class label-based and enrollment-based approaches. SoundBeam achieves improved performance and enables continuous learning by allowing the system to adapt to new sound classes with minimal examples, making it more flexible and practical for real-world applications.

What Problem Does This Paper Address?

Imagine being at a noisy party and wanting to focus only on the conversation with one person, or in a busy street, wanting to hear only the siren of an approaching ambulance while filtering out other traffic noise. This ability to selectively focus on desired sounds while suppressing others is known as "cocktail party effect" or selective hearing. Computationally, this is addressed by Target Sound Extraction (TSE), which aims to extract specific sound events (SEs) from a mixture of sounds.

Existing TSE methods typically rely on either:

  1. Class labels: These systems use predefined labels (e.g., "dog barking," "siren") to guide the extraction. They achieve high performance for known classes but struggle to adapt to new, unseen sound classes.
  2. Enrollment audio samples: These systems use a short audio clip of the target sound as a "query" to find similar sounds in the mixture. This allows them to handle new sound classes but might lead to suboptimal performance compared to label-based methods for known classes.

The challenge is to create a TSE system that combines the strengths of both approaches: high performance for known classes and the flexibility to learn and extract new sound classes without extensive retraining.

Methods

The paper proposes SoundBeam, a unified TSE framework that leverages a shared sound extraction neural network (NN) and a flexible clue encoder.

  • Sound Extraction NN: This core component takes the audio mixture and a target sound embedding vector as input and outputs the estimated target sound. The authors adapt the Conv-TasNet architecture for this purpose, which operates directly in the time domain. A key innovation here is the "Adaptation layer" that conditions the extraction process on the target sound embedding.
  • Clue Encoder: This module generates the target sound embedding vector. SoundBeam employs a "mixed encoder" that can utilize either:
  • Class Label Encoder: Converts a one-hot vector representing a known class into an embedding.
  • Enrollment Encoder: Processes an enrollment audio sample to generate an embedding for a target sound.
  • Unified Training: SoundBeam is trained using a multi-task approach, alternating between using class label embeddings and enrollment embeddings. This encourages the system to learn a common embedding space, allowing it to generalize better.
  • Few-Shot Adaptation: A crucial aspect of SoundBeam is its ability to learn new SE classes with very few enrollment samples. This is achieved by fine-tuning the embedding vectors for new classes while preserving performance on previously learned classes, preventing "catastrophic forgetting."
  • Handling Inactive Classes: The framework also addresses the scenario where a target sound is not present in the mixture, ensuring the system outputs silence. This is incorporated into the training loss.
  • Multi-Target Extraction: SoundBeam can simultaneously extract multiple target sound events by summing their respective embeddings.

Datasets

The experiments were conducted using several datasets:

  • Simulated Mixtures: Created using the FSD corpora (FSD-Kaggle 2018, FSD50K) by mixing isolated SEs with background noise. Datasets were generated for:
  • Single-target extraction (41 and 61 SE classes).
  • Multi-target extraction (mixtures with 3-5 SEs and 1-3 targets).
  • Few-shot adaptation (using 41 "seen" classes and 20 "new" classes).
  • Real Mixtures: A small dataset of real-world recordings from FSD50K containing two or more labeled SE classes, with varying recording conditions and durations.
  • SEC Dataset: Used for retraining a Sound Event Classification (SEC) model.

Key Insights and Ideas

  • Unifying Approaches: The core insight is that by sharing the sound extraction network and learning a common embedding space, class label-based and enrollment-based TSE can be effectively combined, leading to better performance and flexibility.
  • Few-Shot Learning for Sound: SoundBeam demonstrates that continuous learning of new sound classes is feasible with just a few enrollment examples, significantly reducing the need for massive retraining.
  • Adaptation for Robustness: The few-shot adaptation mechanism not only allows learning new classes but also slightly improves performance on existing classes, suggesting a more robust overall model.
  • Inactive Class Detection: Explicitly training with inactive samples improves the system's ability to output silence when a target sound is absent, a crucial feature for practical applications.
  • Multi-Target Extraction Efficiency: The proposed method for simultaneous multi-target extraction is efficient and robust to variations in the number of sources in the mixture.

Key Results

  • Superior Performance: SoundBeam consistently outperformed both class label-based and enrollment-based TSE models across various scenarios, including single-target extraction, multi-target extraction, and handling inactive classes.
  • Few-Shot Adaptation Gains: The SoundBeam-adapt model achieved slightly better performance than a model trained from scratch on 61 classes, demonstrating the effectiveness of the adaptation strategy. For example, performance improved by about 1 dB with as few as 3 enrollments.
  • Inactive Class Handling: Models trained with inactive samples showed significantly better performance in detecting inactive classes, with AUC values above 0.75 and improved attenuation ratios.
  • Multi-Target Extraction: SoundBeam achieved competitive SDR values for multi-target extraction, outperforming baseline methods.
  • Real-World Potential: Preliminary experiments on real mixtures showed that SoundBeam could extract target sounds, with classification performance (mAP) improving significantly after extraction, suggesting its potential for real-world applications.

Limitations and Future Work

  • Computational Intensity: The current framework relies on computationally intensive network configurations, which may limit its applicability in real-time, low-resource scenarios. Future work could explore causal implementations or more efficient models.
  • Real Mixture Evaluation: While promising, the evaluation on real mixtures is preliminary. The authors suggest that future work should focus on designing simulated training data that better approximates real-world mixing conditions and exploring semi-supervised retraining strategies.
  • Noisy Enrollments: The current few-shot adaptation assumes clean enrollment samples. Future work could investigate how to handle noisy enrollments.
  • Improved Inactive Class Handling: While progress was made, there's still a performance trade-off between active and inactive class extraction, especially with a large number of classes. Future research could focus on mitigating this issue.
  • Larger Scale Datasets: The authors suggest extending investigations to even larger datasets like AudioSet to further improve performance and robustness.

Summary updated

Abstract

In many situations, we would like to hear desired sound events (SEs) while being able to ignore interference. Target sound extraction (TSE) tackles this problem by estimating the audio signal of the sounds of target SE classes in a mixture of sounds while suppressing all other sounds. We can achieve this with a neural network that extracts the target SEs by conditioning it on clues representing the target SE classes. Two types of clues have been proposed, i.e., target SE class labels and enrollment audio samples (or audio queries), which are pre-recorded audio samples of sounds from the target SE classes. Systems based on SE class labels can directly optimize embedding vectors representing the SE classes, resulting in high extraction performance. However, extending these systems to extract new SE classes not encountered during training is not easy. Enrollment-based approaches extract SEs by finding sounds in the mixtures that share similar characteristics to the enrollment audio samples. These approaches do not explicitly rely on SE class definitions and can thus handle new SE classes. In this paper, we introduce a TSE framework, SoundBeam, that combines the advantages of both approaches. We also perform an extensive evaluation of the different TSE schemes using synthesized and real mixtures, which shows the potential of SoundBeam.
BibTeX
@article{delcroix2023soundbeam,
  author = {Marc Delcroix and Jorge Bennasar Vázquez and Tsubasa Ochiai and Keisuke Kinoshita and Yasunori Ohishi and Shoko Araki},
  title = {{SoundBeam: Target Sound Extraction Conditioned on Sound-Class Labels and Enrollment Clues for Increased Performance and Continuous Learning}},
  journal = {IEEE/ACM Trans. ASLP},
  year = {2023},
  doi = {10.1109/taslp.2022.3221000},
}
Figure