Semi-parametric trajectory modelling using temporally varying feature mapping for speech recognition

Khe Chai Sim, Shilin LiuInterspeech 2010 · DOI

Abstract

Standard Hidden Markov Models (HMMs) for speech recognition suffer from the observation independence assumption, failing to capture the continuous, smooth trajectories of speech signals. This paper proposes Temporally Varying Feature Mapping (TVFM), a semi-parametric approach that transforms input features into a "trajectory-suppressed" space. By shifting the complexity of trajectory modeling from the HMM architecture to a flexible feature transformation, the approach achieves significant accuracy gains on standard benchmarks like TIMIT and Wall Street Journal (WSJ0) without the heavy computational cost of explicit trajectory models.

Introduction

The fundamental flaw in standard HMM-based speech recognition is the assumption that speech frames are independent given the current state. While dynamic features (delta and acceleration) are commonly used to mitigate this, the underlying model still treats speech as a sequence of piece-wise constant steps. Explicit Trajectory HMMs have been developed to address this but are often computationally prohibitive, typically limited to rescoring small N-best lists rather than searching for the right words from scratch.

This research introduces TVFM as a means to obtain the benefits of trajectory modeling through feature transformation. The core idea is to "flatten" the complex trajectories of speech signals into a form that standard HMMs can process more effectively, effectively "paving the road" for the existing recognition engine.

Methodology

TVFM utilizes a time-varying mapping function $M(t)$ to transform the original feature sequence. This mapping is defined as a weighted combination of $S$ basis matrices ($M_s$):

$$M(t) = \sum_{s=1}^{S} P(s|t)M_s$$

The temporal weights $P(s|t)$ represent the posterior probabilities of HMM states, which are predicted using a feedforward neural network that considers a long temporal context of speech frames.

Estimation Approaches

Two primary methods for learning the basis matrices are explored:

  1. Frame-based Linear Regression (FLR): A least-squares approach minimizing the discrepancy between mapped features and a target trajectory derived from a Trajectory HMM.
  2. Model-based Linear Regression (MLR): A maximum likelihood approach, similar to Constrained Maximum Likelihood Linear Regression (CMLLR), which optimizes the transforms to maximize the likelihood of the HMM generating the transformed features.

Experimental Results

The proposed methods were evaluated on the TIMIT (phone recognition) and Wall Street Journal (WSJ0, continuous speech recognition) datasets. The systems compared include a baseline HMM (System A), standard CMLLR adaptation (System B), TVFM with FLR (System C), and TVFM with MLR (System D).

Phone Recognition (TIMIT)

On the TIMIT dataset, the Model-based TVFM (System D) consistently outperformed all other configurations across varying numbers of Gaussian components.

Phone Error Rate (PER) on TIMIT for different systems and component counts.

As shown in the table above, System D achieved a PER of 36.40% with 32 components, a significant improvement over the baseline HMM's 39.43% and the CMLLR-adapted System B's 38.01%.

Continuous Speech Recognition (WSJ0)

The performance gains were even more pronounced in word error rate (WER) on the WSJ0 benchmark.

Word Error Rate (WER) on WSJ0 for baseline, CMLLR, and TVFM (MLR) systems.

As shown in the results for the si_dt5a and si_dt5b sets, the TVFM (MLR) approach (System D) reduced the WER from 16.44% (baseline) to 13.45% on the si_dt5a set, representing an 18% relative error reduction.

Conclusion

TVFM provides an efficient and effective framework for trajectory modeling in speech recognition. By suppressing the "wiggliness" of speech trajectories at the feature level, it allows standard HMMs to achieve higher accuracy without the need for expensive explicit trajectory modeling. Future work involves refining the neural network weight generation and integrating these concepts into modern deep learning architectures.

Summary updated

BibTeX
@inproceedings{sim2010semi,
  author = {Khe Chai Sim and Shilin Liu},
  title = {{Semi-parametric trajectory modelling using temporally varying feature mapping for speech recognition}},
  booktitle = {Interspeech},
  year = {2010},
  doi = {10.21437/interspeech.2010-34},
}
Figure