Learning Problem-agnostic Speech Representations from Multiple Self-supervised Tasks

Santiago Pascual, Mirco Ravanelli, Joan Serrà, Antonio Bonafonte, Yoshua Bengio

Abstract

Learning good representations without supervision is still an open issue in machine learning, and is particularly challenging for speech signals, which are often characterized by long sequences with a complex hierarchical structure. This paper proposes an improved self-supervised method, where a single neural encoder is followed by multiple workers that jointly solve different self-supervised tasks. The needed consensus across different tasks naturally imposes meaningful constraints to the encoder, contributing to discover general representations and to minimize the risk of learning superficial ones. Experiments show that the proposed approach can learn transferable, robust, and problem-agnostic features that carry on relevant information from the speech signal, such as speaker identity, phonemes, and even higher-level features such as emotional cues.

Introduction

Deep learning traditionally relies on massive amounts of human-labeled data, which is both expensive and time-consuming to produce. In the domain of speech processing, models trained on specific tasks—such as automatic speech recognition (ASR)—often fail to capture other vital information like emotional cues or speaker identity. Furthermore, speech is a high-dimensional signal with a complex hierarchy (phonemes, syllables, words), making it difficult for a single self-supervised task to capture all layers of meaning.

To address these challenges, we introduce the Problem-Agnostic Speech Encoder (PASE). PASE aims to create a "universal" speech feature extractor that discovers robust, general-purpose features without requiring task-specific labels during initial training.

Methods: The PASE Architecture

The core of the PASE framework is a single neural encoder followed by multiple "workers" that jointly solve different self-supervised tasks. This multi-task approach forces the encoder to find a consensus representation that satisfies diverse constraints.

  1. The Encoder: A fully convolutional network that processes raw audio waveforms. The first layer utilizes SincNet, a specialized layer that learns band-pass filters to mimic human auditory perception. The encoder compresses the audio into a 100-dimensional embedding every 10ms.
  2. The Workers: Seven small neural networks employ the encoder's output to solve specific tasks:
  3. Regression Workers: These reconstruct the original waveform, predict the Log Power Spectrum (LPS), calculate MFCCs, and estimate prosody (pitch, energy).
  4. Discrimination Workers: These use contrastive learning to distinguish related from unrelated audio chunks, including Local Info Max (LIM), Global Info Max (GIM), and Sequence Predicting Coding (SPC).
The PASE architecture featuring a SincNet-based encoder and multiple self-supervised workers.

Experimental Setup

The model was pre-trained using a subset of the LibriSpeech dataset (clean English speech). To evaluate the transferability of the learned features, we tested PASE on several downstream tasks:

  • Speaker-ID: Using the VCTK dataset (109 speakers).
  • Emotion Classification: Using the INTERFACE dataset.
  • ASR: Using TIMIT for phoneme recognition and DIRHA for noisy, real-world environments.

Key Results

PASE demonstrated significant improvements over standard hand-crafted features like MFCCs and FBANKs across all evaluation metrics. Notably, the model achieved 97.7% accuracy in emotion recognition, outperforming reported human-level performance (80%). In speaker identification, the fine-tuned PASE model reached 99.3% accuracy.

Classification accuracy across different tasks and backends, comparing PASE to standard features.

Beyond clean audio, PASE showed remarkable implicit robustness. In the noisy DIRHA environment, PASE achieved a Word Error Rate (WER) of 29.8%, compared to 35.8% for MFCCs. Ablation studies confirmed that every worker contributes to the final performance; removing any single task, such as prosody or MFCC prediction, resulted in performance drops for specific downstream applications.

Discussion and Conclusion

The success of PASE highlights the power of multi-task consensus in self-supervised learning. By avoiding reliance on a single objective, the model learns deeper, more transferable abstractions from raw audio. The encoder is designed to be fully parallelizable and easily exportable as a drop-in replacement for traditional features.

Future work will address the current 150ms receptive field limitation to better capture long-term linguistic structures. Additionally, we aim to transition toward entirely "target-free" self-supervision, removing the need for hand-crafted targets like MFCCs in the worker tasks.

Summary updated

Abstract

Learning good representations without supervision is still an open issue in machine learning, and is particularly challenging for speech signals, which are often characterized by long sequences with a complex hierarchical structure. Some recent works, however, have shown that it is possible to derive useful speech representations by employing a self-supervised encoder-discriminator approach. This paper proposes an improved self-supervised method, where a single neural encoder is followed by multiple workers that jointly solve different self-supervised tasks. The needed consensus across different tasks naturally imposes meaningful constraints to the encoder, contributing to discover general representations and to minimize the risk of learning superficial ones. Experiments show that the proposed approach can learn transferable, robust, and problem-agnostic features that carry on relevant information from the speech signal, such as speaker identity, phonemes, and even higher-level features such as emotional cues. In addition, a number of design choices make the encoder easily exportable, facilitating its direct usage or adaptation to different problems.
BibTeX
@inproceedings{pascual2019learning,
  author = {Santiago Pascual and Mirco Ravanelli and Joan Serrà and Antonio Bonafonte and Yoshua Bengio},
  title = {{Learning Problem-agnostic Speech Representations from Multiple Self-supervised Tasks}},
  booktitle = {Interspeech},
  year = {2019},
  url = {https://openreview.net/forum?id=HcIWdQyLBe5},
}
Figure