Towards a Common Speech Analysis Engine

Hagai Aronowitz, Itai Gat, Edmilson da Silva Morais, Weizhong Zhu, Ron Hoory

Abstract

Recent innovations in self-supervised representation learning have led to remarkable advances in natural language processing. That said, in the speech processing domain, self-supervised representation learning-based systems are not yet considered state-of-the-art. We propose leveraging recent advances in self-supervised-based speech processing to create a common speech analysis engine. Such an engine should be able to handle multiple speech processing tasks, using a single architecture, to obtain state-of-the-art accuracy. The engine must also enable support for new tasks with small training datasets. Beyond that, a common engine should be capable of supporting distributed training with client in-house private data. We present the architecture for a common speech analysis engine based on the HuBERT self-supervised speech representation. Based on experiments, we report our results for language identification and emotion recognition on the standard evaluations NIST-LRE 07 and IEMOCAP. Our results surpass the state-of-the-art performance reported so far on these tasks. We also analyzed our engine on the emotion recognition task using reduced amounts of training data and show how to achieve improved results.

Introduction

In traditional speech processing, the development of specialized models has often led to architectural "silos." Separate, bespoke models are typically designed for distinct tasks such as language identification (LID) or speech emotion recognition (SER). This approach presents three significant challenges: the scarcity of labeled data for niche tasks, the high maintenance overhead of managing multiple architectures, and the difficulty of training large-scale models on private, distributed datasets.

While Natural Language Processing (NLP) has largely solved these issues through unified backbones like BERT, speech processing has lagged behind. Speech is a continuous signal lacking the discrete boundaries of text, making the application of predictive "fill-in-the-blank" training more complex. This research introduces a common speech analysis engine that leverages self-supervised learning to create a single, high-performance backbone capable of ruling diverse speech tasks.

Architecture and Methodology

The proposed engine utilizes a two-part architecture designed for both power and simplicity:

The Upstream Backbone

The "brain" of the engine is HuBERT (Hidden-Unit BERT), a self-supervised model. HuBERT is trained by masking portions of an audio signal and predicting "hidden units" (acoustic clusters) within those regions. This process allows the model to learn deep acoustic and linguistic structures from massive amounts of unlabeled audio data without requiring manual transcriptions.

The Downstream Specialist

In a departure from complex, task-specific networks (like deep CNNs or LSTMs), the researchers employed a minimalist downstream setup: temporal average pooling followed by a single linear layer. This design tests the hypothesis that a sufficiently powerful representation makes complex downstream architectures redundant.

Training Strategies

The engine was evaluated using two primary strategies:

  1. Frozen: The HuBERT backbone remains static while only the linear layer is trained.
  2. Finetuned: The entire model, including the backbone, is updated for the specific task.

To address the instability often encountered when training on small or noisy datasets (such as emotion recognition), the team implemented Stochastic Weight Averaging (SWA). By averaging the weights of top-performing models across different training epochs, the engine achieves a more robust and generalized final state.

Experimental Setup

The engine was evaluated across several rigorous benchmarks:

  • NIST-LRE 07: The industry standard for Spoken Language Identification, featuring 14 languages.
  • VoxForge: An open-source LID dataset used for cross-framework comparison.
  • IEMOCAP: The primary benchmark for Speech Emotion Recognition, consisting of approximately 12 hours of emotional speech.
  • Pre-training: The "Base" model was pre-trained on 960 hours of English (LibriSpeech), while the "Large" model utilized 60,000 hours (Libri-light).

Key Results and Insights

State-of-the-Art Performance

The common engine significantly outperformed existing specialized models across all tasks:

  • Language Identification: On NIST-LRE 07 (30s task), the engine achieved an Equal Error Rate (EER) of 0.4%, nearly halving the previous state-of-the-art of 0.7%. On VoxForge, it reached an accuracy of 99.8%.
  • Emotion Recognition: The engine achieved 75.2% accuracy on IEMOCAP, surpassing the previous SOTA of 71.8%. Notably, this was achieved without the complex attention mechanisms or multi-modal (text + audio) inputs often required by other high-performing models.

The Impact of Finetuning and SWA

Because the HuBERT backbone was pre-trained primarily on English, finetuning proved essential for multilingual tasks. Finetuning the "Large" model resulted in a 17% reduction in error rates compared to the "Base" model. Furthermore, SWA provided a "free lunch" in performance for emotion recognition, boosting accuracy from 73.0% to 75.2% without increasing computational costs during inference.

Efficiency

Despite its depth, the HuBERT Large model is highly optimized for production, running 70 times faster than real-time on a standard V100 GPU.

Conclusion

This research demonstrates that a single, self-supervised backbone can replace multiple specialized architectures while achieving superior results. The success of the simple linear downstream layer suggests that the quality of the representation is the primary driver of performance in modern speech processing.

Future work will focus on pre-training the backbone on more diverse, multilingual data and expanding the engine to support tasks like Automatic Speech Recognition (ASR). Additionally, the use of "frozen" representations offers a promising path for distributed training on private data, ensuring high performance without compromising data security.

Summary updated

Abstract

Recent innovations in self-supervised representation learning have led to remarkable advances in natural language processing. That said, in the speech processing domain, self-supervised representation learning-based systems are not yet considered state-of-the-art. We propose leveraging recent advances in self-supervised-based speech processing to create a common speech analysis engine. Such an engine should be able to handle multiple speech processing tasks, using a single architecture, to obtain state-of-the-art accuracy. The engine must also enable support for new tasks with small training datasets. Beyond that, a common engine should be capable of supporting distributed training with client in-house private data. We present the architecture for a common speech analysis engine based on the HuBERT self-supervised speech representation. Based on experiments, we report our results for language identification and emotion recognition on the standard evaluations NIST-LRE 07 and IEMOCAP. Our results surpass the state-of-the-art performance reported so far on these tasks. We also analyzed our engine on the emotion recognition task using reduced amounts of training data and show how to achieve improved results.
BibTeX
@inproceedings{aronowitz2022towards,
  author = {Hagai Aronowitz and Itai Gat and Edmilson da Silva Morais and Weizhong Zhu and Ron Hoory},
  title = {{Towards a Common Speech Analysis Engine}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=r417WSudjw},
}
Figure