Bayesian Neural Network Language Modeling for Speech Recognition

Boyang Xue, Shoukang Hu, Junhao Xu, Mengzhe Geng, Xunying Liu, Helen Meng

Introduction

Modern Automatic Speech Recognition (ASR) systems rely heavily on Neural Network Language Models (NNLMs), such as LSTMs and Transformers, to predict word sequences. While powerful, these models are often over-parameterized, containing millions of weights while frequently being trained on limited domain-specific data. Standard training typically employs point estimation, where each weight is assigned a single fixed value. This approach makes models prone to overfitting and "overconfidence," as they fail to account for the inherent uncertainty in their own parameters or structure.

To address these limitations, this research introduces a comprehensive Bayesian framework designed to model uncertainty and improve generalization in state-of-the-art language models.

A Unified Bayesian Framework for NNLMs

The proposed framework models three distinct types of uncertainty to create more robust language models:

  1. Bayesian NNLMs: These model uncertainty over model parameters (weights) by treating them as Gaussian distributions rather than fixed values.
  2. Gaussian Process (GP) NNLMs: These model uncertainty over neural activations. Instead of a single fixed activation function, the model learns an interpolation of multiple functions (e.g., Sigmoid, Tanh, ReLU, GELU) with associated uncertainty.
  3. Variational NNLMs: These model uncertainty over the hidden layer representations themselves.
Overview of Bayesian, GP, and Variational extensions for LSTM and Transformer architectures.

Full Bayesian learning is traditionally computationally prohibitive for deep architectures like Transformers. To maintain efficiency, the authors utilized Differentiable Architecture Search (DARTS) to selectively apply Bayesian learning. This system automatically identifies which specific gates in LSTMs or layers in Transformers benefit most from Bayesian treatment.

A key insight from the NAS results is that uncertainty is not uniform across the network. In Transformers, the lower layers (closer to the input) typically exhibit higher uncertainty and benefit significantly more from Bayesian modeling than the top layers. This selective application allows the models to remain computationally manageable, with training times only 5–10% slower than standard deterministic models.

Architecture weights for Bayesian vs. fixed components across Transformer layers, showing higher Bayesian preference in lower layers.

Experimental Results

The framework was evaluated on two challenging ASR tasks: the AMI Meeting Corpus (100 hours of multi-microphone meeting audio) and the Oxford-BBC LipReading Sentences 2 (LRS2) (featuring 85% overlapped speech).

The Bayesian framework consistently outperformed standard models and traditional regularization techniques like Dropout:

  • LRS2 (Overlapped Speech): The models achieved a 1.3% absolute (12.1% relative) reduction in Word Error Rate (WER) for LSTMs and a 1.2% absolute (11.3% relative) reduction for Transformers.
  • AMI (Meetings): Significant improvements were noted across all microphone setups. For the multi-distant microphone (mdm8) condition, the Bayesian Transformer reduced WER from 27.0% to 26.8%.
  • Perplexity: In all cases, Bayesian models achieved lower perplexity, indicating superior generalization to unseen text.

Conclusion

By selectively applying Bayesian principles through architecture search, this framework provides a mathematically grounded way to handle model uncertainty in ASR. While scaling these methods to "billion-word" large language models remains a future challenge, the current results demonstrate that modeling uncertainty is critical for robust performance in data-constrained or high-noise environments. Future work will explore applying this framework to fast domain adaptation, where uncertainty modeling is particularly vital.

Summary updated

Abstract

State-of-the-art neural network language models (NNLMs) represented by long short term memory recurrent neural networks (LSTM-RNNs) and Transformers are becoming highly complex. They are prone to overfitting and poor generalization when given limited training data. To this end, an overarching full Bayesian learning framework encompassing three methods is proposed in this paper to account for the underlying uncertainty in LSTM-RNN and Transformer LMs. The uncertainty over their model parameters, choice of neural activations and hidden output representations are modeled using Bayesian, Gaussian Process and variational LSTM-RNN or Transformer LMs respectively. Efficient inference approaches were used to automatically select the optimal network internal components to be Bayesian learned using neural architecture search. A minimal number of Monte Carlo parameter samples as low as one was also used. These allow the computational costs incurred in Bayesian NNLM training and evaluation to be minimized. Experiments are conducted on two tasks: AMI meeting transcription and Oxford-BBC LipReading Sentences 2 (LRS2) overlapped speech recognition using state-of-the-art LF-MMI trained factored TDNN systems featuring data augmentation, speaker adaptation and audio-visual multi-channel beamforming for overlapped speech. Consistent performance improvements over the baseline LSTM-RNN and Transformer LMs with point estimated model parameters and drop-out regularization were obtained across both tasks in terms of perplexity and word error rate (WER). In particular, on the LRS2 data, statistically significant WER reductions up to 1.3% and 1.2% absolute (12.1% and 11.3% relative) were obtained over the baseline LSTM-RNN and Transformer LMs respectively after model combination between Bayesian NNLMs and their respective baselines.
BibTeX
@article{xue2022bayesian,
  author = {Boyang Xue and Shoukang Hu and Junhao Xu and Mengzhe Geng and Xunying Liu and Helen Meng},
  title = {{Bayesian Neural Network Language Modeling for Speech Recognition}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2022},
  url = {https://openreview.net/forum?id=pERO613F5K},
}
Figure