3MAS: a multitask, multilabel, multidataset semi-supervised audio segmentation model

Martin Lebourdais, Pablo Gimeno, Théo Mariotte, Marie Tahon, Alfonso Ortega, Anthony LarcherOdyssey 2024 · DOI

TL;DR

This paper introduces 3MAS, a novel deep learning model for audio segmentation that can simultaneously identify speech, overlap, music, and noise. By framing this as a multilabel problem and employing semi-supervised learning techniques, 3MAS achieves competitive performance across multiple datasets while significantly reducing training time and offering a flexible architecture for future expansion.

What Problem Does This Paper Address?

Audio segmentation, the task of dividing audio signals into meaningful segments, is crucial for many downstream applications like automatic speech recognition (ASR) and speaker diarization. However, real-world audio is complex, containing various sound events like speech, overlapping speech (where multiple people talk at once), music, and noise. Traditionally, these tasks are handled by separate models trained on different datasets, leading to increased computational costs and limited generalization. Furthermore, existing datasets often have incomplete or inconsistent annotations, making it challenging to train robust models for all these events simultaneously. This paper addresses the need for a unified, efficient, and adaptable audio segmentation model that can handle multiple sound classes and diverse, partially annotated datasets.

Methods

The authors propose 3MAS (3-class Multitask, Multilabel, Multidataset Semi-supervised audio segmentation model). The core of their approach involves:

  • Multilabel Segmentation: Instead of treating each sound event as a mutually exclusive class (multiclass), 3MAS frames the problem as a multilabel task. This means multiple labels (speech, overlap, music, noise) can be assigned to the same time frame, reflecting the reality of complex audio.
  • Feature Extraction: They leverage the pre-trained WavLM model as a powerful feature extractor, demonstrating its robustness even for non-speech sounds. They also explore traditional features like MFCCs and chromas, and a more recent learnable feature extractor called LEAF.
  • Sequence Modeling: A Temporal Convolutional Network (TCN) is used for sequence modeling, known for its effectiveness in capturing long-term temporal dependencies in audio.
  • Semi-supervised Learning with Masking: To handle datasets with missing annotations (e.g., overlap present but not labeled), 3MAS employs a clever masking strategy. Unannotated frames are masked during loss calculation, effectively ignoring them and allowing training on heterogeneous data.
  • Data Augmentation: To address class imbalance (especially for music, noise, and overlap), they use two augmentation techniques: adding external music/noise segments and mixing segments within a batch.

Datasets

The paper utilizes several publicly available datasets for training and evaluation:

  • AMI: Primarily used for speech and overlap detection.
  • DIHARD: A challenging dataset for overlapped speech detection, known for its diverse recording conditions.
  • ALLIES: A French meta-corpus for speaker diarization and transcription.
  • Albayzín 2010 & 2012: Datasets for speech, music, and noise segmentation.
  • OpenBMAT: Used for music detection.

The authors create a combined test set by merging partitions from ALLIES, AMI, DIHARD, and Albayzín 2012 to evaluate the model's performance across all target classes.

Key Insights and Ideas

  • Multilabel is Scalable: Framing audio segmentation as a multilabel problem offers better scalability compared to multiclass approaches, especially when dealing with a large number of potential classes.
  • WavLM Generalizes Beyond Speech: The pre-trained WavLM model, primarily trained on speech, shows remarkable generalization capabilities, performing well on music and noise detection tasks.
  • Masking for Heterogeneous Data: The proposed masking strategy is a simple yet effective way to train a single model on multiple datasets with varying annotation availability.
  • Augmentation is Crucial for Under-represented Classes: While speech and overlap detection can perform well with minimal augmentation, music and noise detection significantly benefit from data augmentation techniques.
  • Class Independence Assumption: The multilabel framework assumes class independence, which is a simplification. While effective, it might miss potential correlations between events (e.g., overlap often implies speech).

Key Results

  • Competitive Performance: 3MAS achieves performance comparable to specialized models for each task, with F1-scores reaching up to 98.3% for speech and 78.6% for noise.
  • Training Time Reduction: In certain scenarios (e.g., AMI dataset), the joint VAD+OSD model significantly reduced training time compared to training separate models.
  • Improved Overlap Detection: The multilabel approach, by incorporating information from music and noise, slightly improved overlap detection compared to baselines.
  • Feature Importance: WavLM features consistently provided high detection scores across all classes, outperforming MFCCs and LEAF features in the multilabel setting.

Limitations and Future Work

  • Class Correlation: The paper acknowledges that the multilabel paradigm treats classes as independent, which may not always be true. Future work could explore injecting knowledge of class correlations into the model.
  • Annotation Heterogeneity: The presence of overlap being annotated as noise in some datasets is identified as a limitation that can affect performance.
  • Sequence-Level Understanding: While frame-level predictions are made, further investigation into sequence-level temporal aspects could enhance performance.
  • Domain and Quality Impact: OSD performance is shown to be sensitive to audio quality and speech domain, suggesting that domain adaptation or more robust feature extraction could be beneficial.
  • Extensibility: The authors highlight the model's potential for easy extension to more classes with minimal architectural changes.

Summary updated

BibTeX
@inproceedings{lebourdais2024mas,
  author = {Martin Lebourdais and Pablo Gimeno and Théo Mariotte and Marie Tahon and Alfonso Ortega and Anthony Larcher},
  title = {{3MAS: a multitask, multilabel, multidataset semi-supervised audio segmentation model}},
  booktitle = {Odyssey},
  year = {2024},
  doi = {10.21437/odyssey.2024-33},
}
Figure