Adaptation of deep neural network acoustic models using factorised i-vectors

Penny Karanasou, Yongqiang Wang, Mark J. F. Gales, Philip C. Woodland

Introduction

Deep Neural Networks (DNNs) have significantly advanced Automatic Speech Recognition (ASR), yet they remain challenging to adapt to specific speakers or acoustic environments. Unlike Gaussian Mixture Models (GMMs), the high parameter count of DNNs makes direct adaptation difficult without substantial amounts of new data. A common solution involves using i-vectors—compact, fixed-length representations of audio characteristics—as auxiliary inputs to the network. However, standard i-vectors often conflate speaker identity with environmental noise, leading to sub-optimal performance when a speaker is recorded in a novel, noisy setting. This paper proposes a method to "factorise" these vectors, disentangling speaker characteristics from noise to improve model robustness.

Methodology: Acoustic Factorisation

The authors approach i-vector extraction through the lens of Cluster Adaptive Training (CAT). In this framework, an i-vector is treated as a set of weights used to combine canonical model components. The core innovation of this work is the introduction of Acoustic Factorisation, which involves two key steps:

  1. Two-Part Representation: Instead of a single vector, the system estimates two distinct factors: one for the speaker ($\lambda^{(s)}$) and one for the environment or noise ($\lambda^{(n)}$).
  2. Orthogonality Constraint: To ensure these factors remain distinct, a mathematical constraint is applied during training. Using Lagrange multipliers, the authors enforce orthogonality between the speaker and noise factors, preventing the speaker vector from absorbing noise information and vice versa.

These factorised vectors are then concatenated with standard acoustic features (such as PLP coefficients) and fed into a hybrid DNN-HMM system. This provides the network with explicit, disentangled information about the speech signal, the speaker, and the background environment.

Experimental Setup

The proposed approach was evaluated using a noise-corrupted version of the Wall Street Journal (WSJ) corpus. The training set consisted of approximately 36,500 utterances from 284 speakers, augmented with six types of noise at Signal-to-Noise Ratios (SNRs) between 10dB and 20dB. Testing was conducted across four sets (wsj0_dt, wsj1_dt, wsj0_et, wsj1_et) with SNRs ranging from 5dB to 15dB. Crucially, there was no overlap between training and test speakers.

Results and Discussion

The factorised i-vector approach (Hybrid-fiv) demonstrated consistent improvements over both the baseline DNN and the standard i-vector adaptation (Hybrid-iv).

Word Error Rate (WER) comparison across different test sets for baseline, standard i-vector, and factorised i-vector systems.

As shown in the results, the baseline system achieved a 12.0% WER on the wsj1_dt set. While standard i-vectors reduced this to 11.5%, the factorised approach further lowered the WER to 11.1%. Interestingly, in some cases like the wsj0_et set, standard i-vectors actually degraded performance compared to the baseline (6.4% vs 6.2%), whereas the factorised approach maintained a gain (5.8%). This suggests that "dirty" i-vectors containing noise information can confuse the DNN, while factorisation provides a more reliable adaptation signal.

Furthermore, the modularity of the factorised approach allows the system to generalize to "unseen" speaker-environment pairs. By combining a speaker vector from one recording with a noise vector from another, the system can effectively handle novel acoustic conditions.

Performance of the factorised system on held-out speaker-noise combinations compared to the baseline.

In experiments specifically targeting these held-out combinations, the factorised approach reduced the error rate from 9.6% to 8.3%, confirming its superior generalization capabilities.

Conclusion

By mathematically separating speaker and noise components through an orthogonality constraint, this research provides a robust framework for DNN adaptation in noisy environments. The factorised i-vector approach consistently outperforms standard adaptation techniques, particularly in its ability to handle unseen acoustic conditions. Future work will explore the integration of statistical smoothing (priors) and the application of this factorisation logic to related tasks such as speaker verification and diarization.

Summary updated

BibTeX
@inproceedings{karanasou2014adaptation,
  author = {Penny Karanasou and Yongqiang Wang and Mark J. F. Gales and Philip C. Woodland},
  title = {{Adaptation of deep neural network acoustic models using factorised i-vectors}},
  booktitle = {Interspeech},
  year = {2014},
  doi = {10.21437/interspeech.2014-488},
}
Figure