Speech recognition with speech synthesis models by marginalising over decision tree leaves

John Dines, Lakshmi Saheer, Hui LiangInterspeech 2009

TL;DR

This paper proposes a novel method to create speech recognition (ASR) models using existing speech synthesis (TTS) models. By "marginalizing" over TTS-specific contexts, they enable a unified framework for both ASR and TTS, paving the way for more seamless speech-to-speech translation and speaker adaptation. While initial results show a performance drop in ASR compared to traditional methods, the approach demonstrates feasibility and opens doors for future research in shared acoustic modeling.

What Problem Does This Paper Address?

The fields of Automatic Speech Recognition (ASR) and Text-to-Speech Synthesis (TTS) have historically developed independently, despite sharing similar technological trajectories (rule-based to statistical models). This separation creates a barrier to creating unified systems, particularly for applications like speech-to-speech translation where ASR and TTS components need to work together seamlessly. A key challenge is the differing ways ASR and TTS model context. TTS requires broad contexts to accurately synthesize prosody (duration, pitch), while ASR typically uses more constrained contexts (like triphones) to manage computational complexity. This paper aims to bridge this gap by enabling ASR to leverage TTS models by reconciling these contextual differences.

Methods

The core of the proposed method lies in a technique called "marginalizing over decision tree leaves." Both ASR and TTS systems commonly use decision trees for parameter sharing in their acoustic models. These trees cluster HMM states based on phonetic and supra-segmental contexts.

The authors' approach leverages the fact that TTS models often have a richer set of contexts than ASR. They propose to derive ASR models from these TTS models by effectively ignoring the TTS contexts that are not relevant to ASR. This is achieved by summing over the leaves of the TTS decision tree, which implicitly marginalizes out the non-ASR contexts. This process allows ASR models to be built using the same underlying acoustic parameters as the TTS models, without modifying the original TTS models.

Datasets

  • Wall Street Journal (WSJO) short term speaker training data (SI84): Used for training the acoustic models.
  • November 93 CSR evaluations (5k vocabulary hub task): Used for evaluating ASR performance.

The paper notes that full context labels were generated using a TTS front-end based on the Unisyn lexicon. A portion of the training data was discarded due to alignment issues between forced alignments and full context labels.

Key Insights and Ideas

  • Unified ASR/TTS Framework: The primary insight is the feasibility of creating ASR models directly from TTS models, enabling a shared acoustic modeling framework. This is a significant step towards truly unified speech processing systems.
  • Marginalization as a Bridge: The proposed "marginalizing over decision tree leaves" technique offers a mathematically sound way to reconcile the differing contextual requirements of ASR and TTS.
  • Speaker Adaptation Potential: By enabling shared models, this approach directly supports the application of speaker adaptation techniques across both ASR and TTS, which is crucial for personalized speech translation.
  • Trade-off Between Context and Performance: The initial results highlight a trade-off: while the marginalized models can be derived from rich TTS contexts, this can lead to a degradation in ASR performance compared to standard, specialized ASR models.

Key Results

  • Feasible Approach: The proposed method successfully generates ASR models from TTS models.
  • ASR Performance Degradation: Compared to standard triphone ASR models, the marginalized triphone models showed a performance degradation. For example, in one experiment, the Word Error Rate (WER) increased from 7.1% to 14.1% when using marginalized full-context models compared to standard triphone models.
  • Increased Model Complexity: The marginalized models, when derived from TTS full-context models, resulted in ASR state-clusters with a significantly higher average number of mixture components.
  • Potential for Improvement: When the number of mixture components was increased to match the total parameters of standard models, the marginalized system also improved substantially, though still resulting in a high number of mixture components per state.

Limitations and Future Work

  • ASR Performance Degradation: The most significant limitation acknowledged is the drop in ASR performance compared to standard ASR models. The authors hypothesize this is due to increased data partitioning by the decision tree and confusion between models due to extensive parameter sharing.
  • Excessive Mixture Components: The method can lead to an excessive number of mixture components per state-cluster in the ASR models, which might negatively impact TTS quality.
  • Future Directions:
    • Finding ways to increase ASR model capacity without negatively impacting TTS performance or leading to an excessive number of mixture components.
  • Further analyzing the effects of using non-triphone questions for data partitioning in ASR.
  • Investigating the use of TTS labels in ASR that include prosodic and supra-segmental contexts, potentially incorporating features like log F0 and aperiodicity.
  • Extending studies to analyze TTS performance within this unsupervised adaptation context.

Summary updated

BibTeX
@inproceedings{dines2009speech,
  author = {John Dines and Lakshmi Saheer and Hui Liang},
  title = {{Speech recognition with speech synthesis models by marginalising over decision tree leaves}},
  booktitle = {Interspeech},
  year = {2009},
}

Loading related content...

Figure