Towards A Common Speech Analysis Engine

Hagai Aronowitz, Itai Gat, Edmilson da Silva Morais, Weizhong Zhu, Ron Hoory

TL;DR

Researchers from IBM and Technion have developed a "common speech analysis engine" that uses a single self-supervised backbone (HuBERT) to achieve state-of-the-art results across diverse tasks like language identification and emotion recognition. By combining powerful pre-trained representations with simple downstream classifiers and weight-averaging techniques, they significantly outperformed existing specialized models while requiring less task-specific architecture tuning.

What Problem Does This Paper Address?

In the world of speech processing, we usually build "silos." If you want to identify a language, you build one model; if you want to detect emotion, you build another. This is inefficient for three main reasons:

  1. Data Scarcity: Many specialized tasks don't have enough labeled data to train deep models from scratch.
  2. Maintenance Overhead: Managing separate architectures for every task is a development nightmare.
  3. Privacy: Clients often have small, private datasets that can't be easily used to train massive, data-hungry models.

While Natural Language Processing (NLP) solved this using models like BERT, speech is harder. Unlike text, speech is a continuous signal without discrete "words" or clear boundaries, making it difficult to apply the predictive "fill-in-the-blank" training styles that made NLP models so successful.

Methods

The authors propose a unified architecture consisting of two parts:

  • Upstream (The Brain): They use HuBERT (Hidden-Unit BERT), a self-supervised model. HuBERT learns by masking parts of an audio signal and trying to predict "hidden units" (clusters of sound) in those masked regions. This allows it to learn deep acoustic and linguistic structures from massive amounts of unlabeled audio.
  • Downstream (The Specialist): Instead of complex task-specific networks, they use a surprisingly simple setup: temporal average pooling followed by a single linear layer.

The paper explores two training strategies:

  1. Frozen: Keeping the HuBERT "brain" static and only training the linear layer.
  2. Finetuned: Updating the entire model.

To handle the noise and instability often found in small datasets (like emotion recognition), they employed Stochastic Weight Averaging (SWA). Instead of just picking the "best" model from a single training epoch, they average the weights of several top-performing models from different epochs to create a more robust final version.

Datasets

The engine was evaluated on three major benchmarks:

  • NIST-LRE 07: The standard for Spoken Language Identification (LID), featuring 14 languages.
  • VoxForge: An open-source LID dataset used to compare against the TRILL representation framework.
  • IEMOCAP: The most widely used dataset for Speech Emotion Recognition (SER), containing roughly 12 hours of acted emotional speech.
  • Pre-training data: The "Base" model was pre-trained on 960 hours of English (LibriSpeech), while the "Large" model used 60,000 hours (Libri-light).

Key Insights and Ideas

  • Finetuning is Essential: Because HuBERT was pre-trained primarily on English, the authors found that finetuning was necessary to achieve SOTA results in multilingual tasks. Finetuning the "Large" model reduced the error rate by 17% compared to the "Base" model.
  • The Power of Simplicity: The fact that a simple linear layer on top of HuBERT outperforms complex, task-specific CNNs and LSTMs suggests that the representation is doing the heavy lifting. We may no longer need bespoke architectures for every speech task.
  • Weight Averaging as a "Free Lunch": In emotion recognition, averaging model weights provided a significant boost (from 73.0% to 75.2% accuracy) without adding any computational cost to the final inference time. This is particularly effective when training data is limited and "noisy."

Key Results

The "Common Engine" didn't just work; it dominated:

  • Language Identification (LID): On the NIST-LRE 07 (30s task), it achieved an Equal Error Rate (EER) of 0.4%, a massive improvement over the previous best of 0.7%. On VoxForge, it reached 99.8% accuracy.
  • Emotion Recognition (SER): It achieved 75.2% accuracy on IEMOCAP, surpassing the previous state-of-the-art (71.8%) which often relied on complex attention mechanisms or multi-modal (audio + text) inputs.
  • Efficiency: Despite its power, the HuBERT Large model runs 70 times faster than real-time on a standard V100 GPU.

Limitations and Future Work

The authors acknowledge that while the model is powerful, it was pre-trained on English-only data. Future work involves exploring how pre-training on diverse, multilingual data might further improve the engine. They also aim to expand the engine to handle more tasks (like ASR) and integrate text input to create a truly universal speech/language processing unit. Finally, they highlight the potential for using "frozen" representations to enable distributed training on private data without compromising security.

Summary updated

Abstract

Recent innovations in self-supervised representation learning have led to remarkable advances in natural language processing. That said, in the speech processing domain, self-supervised representation learning-based systems are not yet considered state-of-the-art.We propose leveraging recent advances in self-supervised-based speech processing to create a common speech analysis engine. Such an engine should be able to handle multiple speech processing tasks, using a single architecture, to obtain state-of-the-art accuracy. The engine must also enable support for new tasks with small training datasets. Beyond that, a common engine should be capable of supporting distributed training with client in-house private data.We present the architecture for a common speech analysis engine based on the HuBERT self-supervised speech representation. Based on experiments, we report our results for language identification and emotion recognition on the standard evaluations NIST-LRE 07 and IEMOCAP. Our results surpass the state-of-the-art performance reported so far on these tasks.We also analyzed our engine on the emotion recognition task using reduced amounts of training data and show how to achieve improved results.
BibTeX
@inproceedings{aronowitz2022towards,
  author = {Hagai Aronowitz and Itai Gat and Edmilson da Silva Morais and Weizhong Zhu and Ron Hoory},
  title = {{Towards A Common Speech Analysis Engine}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=r417WSudjw},
}

Loading related content...

Figure