Introduction
Modern Automatic Speech Recognition (ASR) systems rely heavily on Neural Network Language Models (NNLMs), such as LSTMs and Transformers, to predict word sequences. While powerful, these models are often over-parameterized, containing millions of weights while frequently being trained on limited domain-specific data. Standard training typically employs point estimation, where each weight is assigned a single fixed value. This approach makes models prone to overfitting and "overconfidence," as they fail to account for the inherent uncertainty in their own parameters or structure.
To address these limitations, this research introduces a comprehensive Bayesian framework designed to model uncertainty and improve generalization in state-of-the-art language models.
A Unified Bayesian Framework for NNLMs
The proposed framework models three distinct types of uncertainty to create more robust language models:
- Bayesian NNLMs: These model uncertainty over model parameters (weights) by treating them as Gaussian distributions rather than fixed values.
- Gaussian Process (GP) NNLMs: These model uncertainty over neural activations. Instead of a single fixed activation function, the model learns an interpolation of multiple functions (e.g., Sigmoid, Tanh, ReLU, GELU) with associated uncertainty.
- Variational NNLMs: These model uncertainty over the hidden layer representations themselves.
Efficient Learning via Neural Architecture Search
Full Bayesian learning is traditionally computationally prohibitive for deep architectures like Transformers. To maintain efficiency, the authors utilized Differentiable Architecture Search (DARTS) to selectively apply Bayesian learning. This system automatically identifies which specific gates in LSTMs or layers in Transformers benefit most from Bayesian treatment.
A key insight from the NAS results is that uncertainty is not uniform across the network. In Transformers, the lower layers (closer to the input) typically exhibit higher uncertainty and benefit significantly more from Bayesian modeling than the top layers. This selective application allows the models to remain computationally manageable, with training times only 5–10% slower than standard deterministic models.
Experimental Results
The framework was evaluated on two challenging ASR tasks: the AMI Meeting Corpus (100 hours of multi-microphone meeting audio) and the Oxford-BBC LipReading Sentences 2 (LRS2) (featuring 85% overlapped speech).
The Bayesian framework consistently outperformed standard models and traditional regularization techniques like Dropout:
- LRS2 (Overlapped Speech): The models achieved a 1.3% absolute (12.1% relative) reduction in Word Error Rate (WER) for LSTMs and a 1.2% absolute (11.3% relative) reduction for Transformers.
- AMI (Meetings): Significant improvements were noted across all microphone setups. For the multi-distant microphone (mdm8) condition, the Bayesian Transformer reduced WER from 27.0% to 26.8%.
- Perplexity: In all cases, Bayesian models achieved lower perplexity, indicating superior generalization to unseen text.
Conclusion
By selectively applying Bayesian principles through architecture search, this framework provides a mathematically grounded way to handle model uncertainty in ASR. While scaling these methods to "billion-word" large language models remains a future challenge, the current results demonstrate that modeling uncertainty is critical for robust performance in data-constrained or high-noise environments. Future work will explore applying this framework to fast domain adaptation, where uncertainty modeling is particularly vital.
Summary updated