A factorized representation of FMLLR transform based on QR-decomposition

Shakti P. Rath, Martin Karafiát, Ondřej Glembek, Jan ČernockýInterspeech 2012

TL;DR

This paper introduces QR-FMLLR, a novel, factorized representation of the Feature-space Maximum Likelihood Linear Regression (FMLLR) transform. By decomposing the transform into a product of single-parameter matrices, QR-FMLLR offers a more flexible and potentially data-efficient approach to speaker adaptation in speech recognition, achieving comparable performance to standard FMLLR.

What Problem Does This Paper Address?

Speaker adaptation is crucial for improving the performance of large vocabulary continuous speech recognition (LVCSR) systems across different speakers. Existing methods like Maximum Likelihood Linear Regression (MLLR) and its feature-space variant, FMLLR, transform acoustic model parameters or feature vectors to account for speaker variability. However, standard FMLLR estimation can be computationally intensive, often requiring numerical optimization techniques like gradient ascent and matrix inversions. This paper addresses the need for a more efficient and potentially more adaptable FMLLR representation that can handle varying amounts of speaker-specific data.

Methods

The core contribution is the QR-FMLLR representation, which factorizes the FMLLR transform matrix (A) into a product of N² single-parameter matrices. This factorization is inspired by QR-decomposition and involves:

  1. QR Decomposition: Expressing the transform matrix A as a product of an orthogonal matrix (Q) and an upper triangular matrix (R).
  2. Givens Rotations for Q: Decomposing the orthogonal matrix Q into a product of N(N-1) Givens rotations. Each Givens rotation is a sparse, full-rank matrix involving a single parameter (an angle) and operates on a 2D plane of the feature vector.
  3. Diagonal and Purely Upper Triangular Matrices for R: Decomposing the upper triangular matrix R into a product of a diagonal matrix (D) and a purely upper triangular matrix (R). The diagonal matrix D has N single-parameter factors, each scaling a single feature component. The purely upper triangular matrix R is further decomposed into N(N-1)/2 single-parameter factors, similar to Givens rotations, operating on 2D planes.

The estimation of these parameters is performed using an Expectation-Maximization (EM) algorithm. A key advantage is that the ML estimation for QR-FMLLR does not require numerical optimization techniques like gradient ascent or matrix inversions, and it avoids computing matrix determinants. The paper also proposes a sequential estimation scheme where factors are introduced one by one, and parameters are estimated iteratively.

Datasets

The experiments were conducted using the following datasets:

  • Training: The speaker-independent (SI) baseline model was trained on the ctstrain-04 training set, a subset of h5train-03 from the University of Cambridge, comprising approximately 278 hours of speech from Switchboard I, II, and Call Home English.
  • Testing: The system was evaluated on the Hub5 Eval-01 test set, which includes 20 conversations from Switchboard-1, Switchboard-2, and Switchboard cellular corpora, totaling over 6 hours of speech.

Key Insights and Ideas

  • Factorization for Efficiency: Decomposing the FMLLR transform into a product of many simple, single-parameter matrices (inspired by QR-decomposition) is a novel way to represent the transform. This factorization leads to a more structured and potentially more efficient estimation process.
  • Elimination of Numerical Optimization: A significant advantage of QR-FMLLR is that its ML estimation bypasses the need for computationally expensive gradient ascent and matrix inversions, which are common in standard FMLLR.
  • Flexibility in Structure: The factorized nature of QR-FMLLR allows for the construction of various LT structures (e.g., full, orthogonal, upper triangular, diagonal, block-diagonal) by selectively including or excluding factors. This offers greater control over model complexity.
  • Potential for Data-Driven Adaptation: The paper conjectures that QR-FMLLR is amenable to speaker adaptation with varying amounts of data, from very short to large. This is a promising direction for adapting to speakers with limited speech samples.

Key Results

The paper presents experimental results on an LVCSR task using 39-dimensional MFCC features.

  • Comparable Performance: For full and block-diagonal matrices, QR-FMLLR achieved performance comparable to standard FMLLR.
  • Improved Performance for Diagonal Case: QR-FMLLR showed better performance than standard FMLLR for the diagonal matrix structure.
  • Analysis of Different Structures: Experiments with various LT structures (Upper Triangular, Orthogonal, Block Diagonal with different configurations) demonstrated that QR-FMLLR can effectively implement these structures. For instance, an Orthogonal + Diagonal transform performed comparably to a Block Diagonal transform. 3-diagonal and 5-diagonal matrices were found to be less effective than Block Diagonal.

Limitations and Future Work

  • Approximation in Givens Rotation Estimation: The estimation of the parameter for Givens rotations involves a small-angle approximation for cosine and sine functions. While this simplifies the math, it means the resulting Q and R matrices may not be exactly orthogonal or purely upper triangular, respectively.
  • Sequential Estimation Complexity: While the sequential estimation is more manageable than simultaneous estimation, it still requires iterating through the data multiple times to estimate all factors. The paper notes that this can be computationally expensive.
  • Future Work: The authors explicitly state that they will investigate the conjecture that QR-FMLLR is amenable to speaker adaptation with varying amounts of speaker-specific data in their future work. This includes exploring how to optimally adjust the LT structure using data-driven methods like Minimum Description Length (MDL).

Summary updated

BibTeX
@inproceedings{rath2012factorized,
  author = {Shakti P. Rath and Martin Karafiát and Ondřej Glembek and Jan Černocký},
  title = {{A factorized representation of FMLLR transform based on QR-decomposition}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure