Latent perceptual mapping: a new acoustic modeling framework for speech recognition

Shiva Sundaram, Jerome R. BellegardaInterspeech 2010

TL;DR

This paper introduces Latent Perceptual Mapping (LPM), a novel acoustic modeling framework for speech recognition that leverages dimensionality reduction to create parsimonious, template-like representations of speech. LPM achieves competitive phoneme classification accuracy comparable to traditional methods, potentially with significant savings in computational costs.

What Problem Does This Paper Address?

Traditional speech recognition heavily relies on Hidden Markov Models (HMMs), which, while dominant, face challenges with model mismatch and parameter explosion, especially with continuous parameter models. Discrete parameter models, on the other hand, suffer from information loss due to quantization. Template-based approaches offer an alternative by directly using speech instances as templates, avoiding HMM limitations. However, these have historically been impractical due to high computational demands. This paper addresses the need for an acoustic modeling framework that combines the benefits of template-based methods (avoiding HMM issues, aligning with perceived acoustic events) with reduced computational complexity, making them more viable for practical applications.

Methods

The core of the proposed framework is Latent Perceptual Mapping (LPM), inspired by Latent Semantic Analysis. The process involves:

  1. Feature Extraction: Acoustic features (e.g., MFCCs) are extracted from speech segments.
  2. Codebook Generation: A set of reference centroids (a codebook) is created by unsupervised clustering of these feature vectors.
  3. Co-occurrence Matrix: Each speech segment is represented as a histogram of how many of its feature vectors fall into each codebook centroid. This forms a "unit-document" matrix.
  4. Dimensionality Reduction (SVD): Singular Value Decomposition (SVD) is applied to this co-occurrence matrix to obtain a reduced-rank approximation. This step identifies a latent perceptual space.
  5. Latent Space Mapping: The segments are mapped to points in this lower-dimensional latent space, forming "acoustic prototypes."

For recognition, a new segment is mapped into this latent space using the learned codebook and singular vectors. Similarity is then computed, and classification is performed using a K-nearest neighbor rule. The key novelty lies in using SVD on a co-occurrence matrix of acoustic units to create a parsimonious, template-like representation in a latent space.

Datasets

  • TIMIT: The standard read speech TIMIT database was used for both training and evaluation.
  • The experiments used the standard NIST training and test sets.
  • 'SA' speakers were excluded.
  • The /q/ phoneme was excluded.
  • The original 61 phoneme classes were reduced to 48 for training and then mapped to 39 for evaluation, consistent with prior work.
  • Training set: 140,225 phoneme segments.
  • Test set: 7,125 phoneme segments.

Key Insights and Ideas

  • Parsimonious Template Representation: LPM offers a way to create template-like representations that are much lower-dimensional than raw acoustic features, making them computationally more efficient.
  • Bridging Template and Parametric Models: The framework aims to capture the strengths of template-based methods (direct alignment with acoustic events, avoiding HMM assumptions) while mitigating their computational burden.
  • Data-Driven Acoustic Prototypes: The acoustic prototypes are derived directly from the data through clustering and SVD, allowing them to naturally reflect perceived acoustic events.
  • Feature Set Impact: The choice of acoustic features significantly impacts performance, with more global temporal information (HAL features) leading to better results than purely local features (STD).
  • Dimensionality Trade-off: While higher dimensionality in the latent space generally improves accuracy, there's a point of diminishing returns, suggesting potential for significant savings in online processing by retaining fewer dimensions.

Key Results

  • Competitive Accuracy: LPM achieved competitive phoneme classification accuracy on the TIMIT database.
  • The best performance for the HAL feature set was 65.19% accuracy.
  • This is comparable to existing non-HMM techniques and early context-independent HMM results (around 64%).
  • Feature Set Performance:
    • LPM+STD: Best accuracy of 56.46% (with ~20% variance retained).
  • LPM+GOL: Best accuracy of 58.07% (with ~20% variance retained).
  • LPM+HAL: Best accuracy of 65.19% (with ~80% variance retained).
  • Codebook Size: A codebook size of 3000-3500 generally yielded the best LPM performance across feature sets.
  • Dimensionality Savings: For HAL and GOL features, performance did not degrade significantly with fewer latent dimensions, indicating potential for computational savings.

Limitations and Future Work

  • Limitations:
    • The paper presents a "pilot study" on context- and speaker-independent phoneme classification.
  • The current work focuses on phoneme classification, not full speech recognition.
  • The performance of the STD feature set is significantly lower than HAL, suggesting that incorporating more global temporal information is crucial.
  • Future Work:
    • Explore different clustering strategies to imbue the STD feature set with segment-level temporal information.
  • Investigate modifications to the K-means procedure for codebook derivation.
  • Incorporate unsupervised context information into the latent representation of acoustic prototypes.
  • Explore the "folding in" approach for mapping new segments more deeply.

Summary updated

BibTeX
@inproceedings{sundaram2010latent,
  author = {Shiva Sundaram and Jerome R. Bellegarda},
  title = {{Latent perceptual mapping: a new acoustic modeling framework for speech recognition}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure