Abstract
Learning good representations without supervision is still an open issue in machine learning, and is particularly challenging for speech signals, which are often characterized by long sequences with a complex hierarchical structure. This paper proposes an improved self-supervised method, where a single neural encoder is followed by multiple workers that jointly solve different self-supervised tasks. The needed consensus across different tasks naturally imposes meaningful constraints to the encoder, contributing to discover general representations and to minimize the risk of learning superficial ones. Experiments show that the proposed approach can learn transferable, robust, and problem-agnostic features that carry on relevant information from the speech signal, such as speaker identity, phonemes, and even higher-level features such as emotional cues.
Introduction
Deep learning traditionally relies on massive amounts of human-labeled data, which is both expensive and time-consuming to produce. In the domain of speech processing, models trained on specific tasks—such as automatic speech recognition (ASR)—often fail to capture other vital information like emotional cues or speaker identity. Furthermore, speech is a high-dimensional signal with a complex hierarchy (phonemes, syllables, words), making it difficult for a single self-supervised task to capture all layers of meaning.
To address these challenges, we introduce the Problem-Agnostic Speech Encoder (PASE). PASE aims to create a "universal" speech feature extractor that discovers robust, general-purpose features without requiring task-specific labels during initial training.
Methods: The PASE Architecture
The core of the PASE framework is a single neural encoder followed by multiple "workers" that jointly solve different self-supervised tasks. This multi-task approach forces the encoder to find a consensus representation that satisfies diverse constraints.
- The Encoder: A fully convolutional network that processes raw audio waveforms. The first layer utilizes SincNet, a specialized layer that learns band-pass filters to mimic human auditory perception. The encoder compresses the audio into a 100-dimensional embedding every 10ms.
- The Workers: Seven small neural networks employ the encoder's output to solve specific tasks:
- Regression Workers: These reconstruct the original waveform, predict the Log Power Spectrum (LPS), calculate MFCCs, and estimate prosody (pitch, energy).
- Discrimination Workers: These use contrastive learning to distinguish related from unrelated audio chunks, including Local Info Max (LIM), Global Info Max (GIM), and Sequence Predicting Coding (SPC).
Experimental Setup
The model was pre-trained using a subset of the LibriSpeech dataset (clean English speech). To evaluate the transferability of the learned features, we tested PASE on several downstream tasks:
- Speaker-ID: Using the VCTK dataset (109 speakers).
- Emotion Classification: Using the INTERFACE dataset.
- ASR: Using TIMIT for phoneme recognition and DIRHA for noisy, real-world environments.
Key Results
PASE demonstrated significant improvements over standard hand-crafted features like MFCCs and FBANKs across all evaluation metrics. Notably, the model achieved 97.7% accuracy in emotion recognition, outperforming reported human-level performance (80%). In speaker identification, the fine-tuned PASE model reached 99.3% accuracy.
Beyond clean audio, PASE showed remarkable implicit robustness. In the noisy DIRHA environment, PASE achieved a Word Error Rate (WER) of 29.8%, compared to 35.8% for MFCCs. Ablation studies confirmed that every worker contributes to the final performance; removing any single task, such as prosody or MFCC prediction, resulted in performance drops for specific downstream applications.
Discussion and Conclusion
The success of PASE highlights the power of multi-task consensus in self-supervised learning. By avoiding reliance on a single objective, the model learns deeper, more transferable abstractions from raw audio. The encoder is designed to be fully parallelizable and easily exportable as a drop-in replacement for traditional features.
Future work will address the current 150ms receptive field limitation to better capture long-term linguistic structures. Additionally, we aim to transition toward entirely "target-free" self-supervision, removing the need for hand-crafted targets like MFCCs in the worker tasks.
Summary updated