TL;DR
This paper investigates using the powerful wav2vec unsupervised learning framework for multiclass audio segmentation, aiming to improve performance over traditional features. While wav2vec excels at segmenting speech-related classes, it struggles with music segmentation, limiting overall gains.
What Problem Does This Paper Address?
Audio segmentation systems aim to label different regions of an audio signal, such as speech, music, or noise. While deep learning has advanced this field, obtaining large amounts of labeled data for training is a significant bottleneck. Unsupervised learning methods, like wav2vec, offer a promising solution by learning representations from vast amounts of unlabeled audio. This paper explores how these learned representations can be applied to multiclass audio segmentation, specifically comparing them against traditional feature sets.
Methods
The core of this work involves using pre-trained wav2vec models as feature extractors for a multiclass audio segmentation task. The wav2vec system itself consists of a convolutional neural network (CNN) encoder followed by a recurrent neural network (RNN) (specifically a Gated Recurrent Unit or GRU). This system is trained using a contrastive predictive coding approach, where it learns to predict future or past audio frames.
For the segmentation task, the pre-trained wav2vec model is frozen, and its output embeddings are used as input to various neural network classifiers. The paper evaluates three classifier architectures:
- 1BLSTM: A single bidirectional LSTM layer.
- 2BLSTM pool: Two bidirectional LSTM layers with an average pooling mechanism in between to smooth temporal outputs.
- 2BLSTM Pool + mixup: The 2BLSTM pool architecture augmented with the mixup technique, which creates new training examples by linearly combining existing ones.
The paper also investigates the impact of data augmentation (adding noise and room impulse responses) during wav2vec pre-training and the effect of using different amounts of training data.
Datasets
- Training for wav2vec: The paper uses a combination of datasets for unsupervised pre-training. These include:
- Albayzín 2018 and 2020 evaluations, and the Spanish partition of Commonvoice 2 for the "wav2vec base" model (approx. 1200 hours).
- For the "wav2vec extended" model, significantly more data was used, including LibriSpeech, LibriLight, Tedlium, VoxCeleb 1&2 (approx. 60,000 hours of speech), and a music dataset (approx. 1700 hours).
- Training and Evaluation for Segmentation: The Albayzín 2010 dataset is used for multiclass audio segmentation. This dataset contains 87 hours of broadcast news audio, divided into training and testing sets. The classes are speech (sp), music (mu), speech over music (sm), speech over noise (sn), and others (ot). The "others" class is not evaluated.
Key Insights and Ideas
- Visualizing Representations: t-SNE visualizations show that wav2vec representations lead to much more separable clusters compared to traditional log Mel and chroma features, suggesting better inherent class discrimination.
- Speech vs. Music Performance: Wav2vec representations significantly improve segmentation performance for classes containing speech (speech, speech-over-music, speech-over-noise).
- Music Segmentation Challenge: A consistent limitation is the degradation in performance for segmenting isolated music, where traditional features outperform wav2vec.
- Impact of Data Scale: Using a larger pre-training dataset ("wav2vec extended") generally leads to better results, especially for speech-related classes.
- Data Augmentation's Role: Data augmentation during wav2vec pre-training shows mixed results, with the "wav2vec base no aug" model sometimes performing better on music and speech-over-music classes.
Key Results
- The best performing system, using the 2BLSTM pool + mixup classifier with wav2vec extended no aug features, achieved a relative improvement of 6.8% compared to the same setup using traditional log Mel and chroma features.
- For speech-related classes, wav2vec representations consistently showed improvements over traditional features.
- However, the music class error rate remained high, limiting the overall average error reduction. For instance, in Table 1, the "wav2vec base" model had a music error of 20.37% compared to 16.28% for traditional features.
Limitations and Future Work
- Music Segmentation: The primary limitation identified is the underperformance of wav2vec representations in segmenting isolated music. This hinders the overall potential gains.
- Future Directions: The authors suggest that future research should focus on developing more robust audio representations that can simultaneously handle speech and music effectively. They point to ongoing research in learning universal audio representations as a promising avenue.
Summary updated