Towards the Next Frontier in Speech Representation Learning Using Disentanglement

Ganapathy SriramarXiv 2024 · DOI · arXiv

TL;DR

This paper introduces "Learn2Diss," a novel self-supervised learning framework for speech that disentangles semantic (what is said) and non-semantic (who is speaking, channel characteristics) information. By jointly training separate frame-level and utterance-level encoders and minimizing their mutual information, Learn2Diss achieves state-of-the-art results on various speech tasks, demonstrating the benefit of learning distinct representations for different speech attributes.

What Problem Does This Paper Address?

Traditional self-supervised learning (SSL) for speech has primarily focused on learning representations from short, frame-level segments of audio, often by masking and predicting parts of the speech signal. While effective for tasks like speech recognition, these methods tend to overlook or conflate information that remains consistent across an entire utterance, such as speaker identity or acoustic channel characteristics. This leads to representations that might be overly influenced by non-semantic factors, potentially hindering performance on tasks requiring pure semantic understanding, or vice-versa. The paper aims to bridge this gap by developing a method that explicitly disentangles these distinct types of information within speech signals.

Methods

The core of the Learn2Diss framework is a dual-encoder architecture. It comprises:

  • Feature Extractor: A frozen, pre-trained model (like HuBERT or wavLM) that processes raw audio into initial representations.
  • Frame-level Encoder (g(.)): This encoder is designed to capture semantic information. It's initialized with pre-trained frame-level models and further trained using a combination of pseudo-phonemic prediction (similar to HuBERT/wavLM) and a "pseudo-con loss" which encourages learning semantic properties.
  • Utterance-level Encoder (h(.)): This encoder focuses on non-semantic information, specifically speaker characteristics. It's initialized with a pre-trained utterance-level model (like ECAPA-TDNN) and trained using contrastive learning (Info-NCE) and clustering losses.
  • Disentanglement Module: The key innovation is the joint training of these two encoders using a mutual information (MI) minimization objective. Specifically, they use the Contrastive Log-ratio Upper Bound (CLUB) to estimate and minimize the MI between the internal representations of the frame-level and utterance-level encoders. This encourages the frame-level encoder to focus on semantic content and the utterance-level encoder to focus on non-semantic attributes, while allowing them to mutually enrich each other.

Datasets

The models were trained on the LibriSpeech dataset (960 hours). Evaluation was performed on various benchmarks, including:

  • SUPERB: A comprehensive benchmark for evaluating speech representations across a wide range of tasks (semantic and non-semantic).
  • ZeroSpeech 2021: Used for acoustic unit discovery tasks.
  • LibriLight (1-hour and 10-hour splits) and Librispeech (100-hour splits): Used for Automatic Speech Recognition (ASR) fine-tuning.

Key Insights and Ideas

  • Explicit Disentanglement is Beneficial: The paper strongly suggests that explicitly separating semantic and non-semantic information leads to more specialized and effective representations for downstream tasks.
  • Mutual Information as a Disentanglement Tool: Using MI minimization is a powerful, unsupervised way to encourage encoders to learn distinct aspects of the data.
  • Frame vs. Utterance Level: The work highlights the importance of considering information at different temporal scales (frame-level for semantics, utterance-level for speaker identity) within a single framework.
  • Plug-in Framework Potential: The authors hypothesize that their disentanglement approach could be applied to various existing SSL models, acting as a "plug-in" to improve their representation quality.

Key Results

  • State-of-the-Art Performance: Learn2Diss achieved state-of-the-art results on several tasks within the SUPERB benchmark, particularly for semantic tasks (e.g., ASR WER improvement of 19.3% with disentanglement) and non-semantic tasks (e.g., speaker identification accuracy improvement of 49.1%).
  • Improved Semantic Representations: The frame-level encoder from Learn2Diss showed significant improvements in semantic tasks like ASR.
  • Improved Non-Semantic Representations: The utterance-level encoder demonstrated enhanced performance on non-semantic tasks like speaker identification and verification.
  • Acoustic Unit Discovery: The models achieved competitive results on the ZeroSpeech 2021 challenge for acoustic unit discovery.
  • ASR Finetuning: Learn2Diss models achieved state-of-the-art or comparable results when fine-tuned for ASR.

Limitations and Future Work

  • Pre-training Overhead: The Learn2Diss framework adds an extra pre-training step, increasing computational cost. The authors suggest exploring early stopping of individual encoders before MI optimization to potentially mitigate this.
  • Single Source of Variability Assumption: The current model assumes a single source of variability in the non-semantic component (e.g., one speaker per utterance). Extending this to multi-speaker audio or music is a future direction.
  • Speech Understanding Focus: The current evaluation is limited to speech understanding tasks. Future work could explore Learn2Diss for generation tasks like voice conversion or speech synthesis.
  • Generalizability: While hypothesized as a plug-in, further research is needed to confirm its applicability to a broader range of frame-level SSL models.

Summary updated

BibTeX
@misc{sriram2024towards,
  author = {Ganapathy Sriram},
  title = {{Towards the Next Frontier in Speech Representation Learning Using Disentanglement}},
  year = {2024},
  eprint = {2407.02543},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.02543},
}
Figure