TL;DR
This paper explores using synthetic speech to create reference templates for template-based Automatic Speech Recognition (ASR). They found that high-quality synthetic speech, when combined with robust posterior features, can achieve performance comparable to using natural speech templates, opening doors for more flexible and scalable ASR systems.
What Problem Does This Paper Address?
Traditional template-based ASR systems rely on collecting a large number of natural speech recordings to create reference templates for each word in the vocabulary. This process is time-consuming, expensive, and makes it difficult to adapt the system to new domains or expand the vocabulary. The authors aim to overcome these limitations by investigating the use of synthetic speech, generated by Text-to-Speech (TTS) systems, as a source for these reference templates. This approach could significantly reduce the need for in-domain data collection and simplify vocabulary expansion.
Methods
The core of the paper lies in combining two key technologies:
- Posterior Features: Instead of using traditional spectral features (like PLP), the system uses phoneme class-conditional probabilities (posterior features) estimated by a Multi-Layer Perceptron (MLP). These features are known to be more robust to variations in speech and generalize better.
- Synthetic Speech Templates: The researchers generated reference templates using a Festival HTS-based TTS system. They then used these synthetic templates within the template-based ASR framework, which employs Dynamic Time Warping (DTW) for matching.
- Weighted Symmetric KL-Divergence: For comparing posterior feature vectors, they used a weighted symmetric Kullback-Leibler (wSKL) divergence, which was found to be more effective than standard distance measures for posterior features.
The novelty lies in the integration of these synthetic templates with posterior features, leveraging the robustness of both to address the data collection bottleneck.
Datasets
- Phonebook Corpus: This corpus of US English read telephone speech was used for both training and evaluation.
- CMU ARCTIC databases: These databases were used to train the HTS-based TTS system.
The experiments were conducted on two tasks: a 75-word task and a 600-word task, using speaker- and task-independent setups.
Key Insights and Ideas
- Synthetic Speech is Viable: The most significant insight is that high-quality synthetic speech can indeed be used to create reference templates that yield performance comparable to natural speech templates, especially when using posterior features.
- Posterior Features are Key: The robustness of posterior features is crucial for the success of synthetic templates. They appear to be less sensitive to the potential imperfections of synthetic speech compared to traditional spectral features.
- Intelligibility Matters: The intelligibility of the synthetic voice directly correlates with the ASR system's performance when using posterior features. More intelligible voices lead to better recognition accuracy.
- No Adaptation Needed (Initially): The study showed that even without adapting the MLP to the synthetic voices, comparable performance could be achieved, highlighting the inherent generalization capabilities of the MLP and posterior features.
Key Results
- Posterior Features Outperform PLP: Across the board, using posterior features consistently yielded much higher word accuracy than using PLP features, both with natural and synthetic templates. For example, on the 75-word task, posterior features achieved up to 98.8% accuracy with natural templates and 98.2% with synthetic templates, while PLP features reached only 65.1% with natural and 67.8% with synthetic.
- Comparable Performance: The best results with synthetic speech templates using posterior features were very close to those obtained with natural speech templates (e.g., 98.2% vs. 98.8% on the 75-word task).
- Voice Quality Impact: The choice of synthetic voice significantly impacted performance. The RMS voice consistently performed best among the tested synthetic voices.
- Multiple Templates: While using multiple synthetic templates per word generally improved performance, the gains were not always substantial, and the best single synthetic voice (RMS) often performed as well as or better than combinations.
Limitations and Future Work
- Quality of TTS: The paper acknowledges that the performance is highly dependent on the quality of the synthetic speech. Future work includes exploring higher-quality TTS systems.
- Posterior Feature Estimation: Adapting the MLP to better handle synthetic speech (e.g., using hierarchical MLPs) is suggested as a way to further improve performance.
- Complementarity of Voices: Investigating the complementarity between natural and synthetic speech templates, and how to best combine different synthetic voices, is identified as an area for future research.
- Generalizability: The authors plan to investigate if these findings generalize to synthetic voices trained on different corpora (e.g., Wall Street Journal).
- Broader ASR Systems: The potential to use synthetic speech for training other ASR systems, like Kullback-Leibler divergence based HMMs (KL-HMMs), is also mentioned.
Summary updated