Abstract
This paper addresses the challenge of overfitting in acoustic modeling for speech recognition when training data is limited. By employing sparse banded precision matrices, the authors propose a "Goldilocks" solution that balances model complexity and generalization. Through $l_1$ regularization and strategic feature interleaving, the resulting models achieve better word error rates (WER) than traditional diagonal models while remaining more computationally efficient than full-covariance systems.
Introduction
Acoustic models in speech recognition often face a trade-off between simplicity and representational power. Diagonal covariance matrices are efficient but ignore feature correlations, while full covariance matrices capture these correlations at the cost of excessive parameters, leading to overfitting in low-resource scenarios. This paper explores the use of sparse banded precision matrices to bridge this gap, creating models that are compact enough to avoid overfitting but flexible enough to capture vital feature correlations.
Methodology
The core of the approach lies in manipulating the precision matrix (the inverse of the covariance matrix) through three primary techniques:
- $l_1$ Regularization: By adding an $l_1$ penalty to the standard Maximum Likelihood Estimation (MLE) objective function, the training process naturally drives non-essential elements of the precision matrix to zero. This induces sparsity, reducing the total number of parameters the model must learn.
- Banded Structure: To optimize for hardware storage and computation, the authors enforce a banded structure. In this configuration, only elements within a specific distance ($k$) from the main diagonal are allowed to be non-zero. This is more efficient than a general sparse matrix because it eliminates the need to store the coordinates of every non-zero element.
- Feature Interleaving: Standard feature layouts (e.g., all MFCCs followed by all deltas) separate highly correlated features. The authors propose interleaving these features ($m_1, \Delta m_1, \Delta\Delta m_1, \dots$) to ensure that the strongest correlations reside near the diagonal, maximizing the effectiveness of the banded structure.
Training and Implementation
The precision matrices are updated iteratively during the Expectation-Maximization (EM) process. The authors compared two training flows: one where the precision matrix is updated within the main training loop and another where it is updated post-hoc after training full covariance models. Interestingly, both methods yielded similar results, leading the authors to favor the simpler post-hoc conversion.
Experimental Results
The approach was evaluated using the WSJ0 (Wall Street Journal) corpus, utilizing approximately 14.5 hours of training audio. Key findings include:
- Superiority over Diagonal Models: A sparse banded model with a half-bandwidth of 4 used only ~780 parameters per HMM state (fewer than the ~936 for a diagonal model) but achieved a lower Word Error Rate (WER) of 6.46% vs 6.71%.
- Efficiency of Interleaving: Using the standard HTK feature layout required a bandwidth of 28 to reach the same accuracy that the proposed interleaved layout achieved with a bandwidth of only 8. This represents a massive saving in computation and memory.
- Competitive Performance: With a half-bandwidth of 8, the banded model achieved a WER of 6.20%, nearly matching the performance of a much more complex, non-banded sparse model (6.07%).
Conclusion
Strategic feature ordering combined with sparse banded precision matrices provides a robust framework for acoustic modeling in low-resource environments. By forcing important information into a narrow diagonal band, researchers can achieve significant model compression and improved generalization without sacrificing the accuracy provided by feature correlation modeling.
Summary updated