TL;DR
This paper introduces a novel approach to improve text-to-speech (TTS) synthesis quality by leveraging same-language machine translation (MT). By generating multiple paraphrased versions of the input text, the TTS system can select the most synthesizable option, overcoming limitations of insufficient speech units. This method significantly enhances synthesis quality by providing the TTS system with a richer set of alternatives.
What Problem Does This Paper Address?
Modern text-to-speech (TTS) systems aim to produce natural-sounding speech from any given text. A significant challenge arises when the TTS system encounters words or phrases for which it lacks appropriate pre-recorded speech units in its database. This can lead to degraded synthesis quality, especially in open-domain scenarios. Traditional approaches often require modifying the TTS system itself to handle alternative representations like word lattices. This paper addresses the need for a more flexible method that can improve TTS quality without requiring deep integration with specific synthesis architectures, allowing for better handling of out-of-vocabulary or difficult-to-synthesize segments.
Methods
The core of this paper is a hybrid system combining machine translation (MT) and unit selection speech synthesis. The key innovation is the use of same-language machine translation (SLMT). Instead of translating between different languages, an MT system is trained to translate English text into alternative English paraphrases that retain the original meaning.
Here's a breakdown of the methodology:
- SLMT Corpus Creation: A parallel corpus of English sentences was created where each pair consisted of a source sentence and a target sentence with the same meaning. This corpus was specifically designed to encourage the MT system to generate phrases that are likely to be present in the ARCTIC speech database, which is commonly used for TTS research.
- Machine Translation System: The MATREX system was employed, which integrates phrase-based statistical machine translation (SMT) with marker-based chunking (from Example-Based Machine Translation - EBMT) to enhance phrase table accuracy. Word alignments were performed using GIZA++, and language models were trained with SRILM. The system was seeded with additional information from the EBMT component.
- N-best List Generation: The trained SLMT system takes the input text and generates an "N-best list" – a ranked set of alternative paraphrased sentences.
- Synthesizer Integration: A unit selection diphone speech synthesizer then receives this N-best list. Crucially, the synthesizer itself evaluates each hypothesis in the list. It uses internal metrics, such as the "total number of joins required" and "total join cost," to estimate how well it can synthesize each alternative.
- Selection: The synthesizer selects the hypothesis from the N-best list that it estimates it can synthesize with the best quality. The original input text is also considered as a candidate, ensuring that if none of the generated paraphrases are better, the original text is used.
Datasets
- ARCTIC Corpus: This publicly available speech database was used as the starting point for creating the SLMT training corpus. The ARCTIC corpus is widely used in speech synthesis research and includes multiple free speech databases.
- Custom SLMT Corpus: A parallel monolingual English corpus was developed for this work. It contained 500 sentence pairs (250 pairs) derived from the ARCTIC corpus. The ARCTIC sentences were consistently used as the target element in each pair to guide the MT system towards generating phrases synthesizable by ARCTIC voices.
- Test Corpus: A test corpus of 30 sentences was created. This corpus intentionally included phrases not present in the synthesizer's speech data to specifically test the system's ability to handle such cases. Some vocabulary from the MT training corpus was also included in the test set.
Key Insights and Ideas
- Leveraging MT for Paraphrasing: The core idea of using same-language machine translation to generate alternative phrasings for TTS is a novel and effective way to address the "out-of-units" problem.
- Synthesizer-Driven Selection: Empowering the synthesizer to choose the best hypothesis from the N-best list based on its own quality metrics (join cost, number of joins) is a key to the system's success. This avoids forcing a potentially suboptimal paraphrase onto the synthesizer.
- Targeting Synthesizer-Specific Vocabulary: Designing the SLMT corpus to favor phrases found in the ARCTIC corpus directly addresses the limitation of insufficient units by generating alternatives that are more likely to be synthesizable.
- Flexibility and Modularity: The approach is designed to be relatively independent of the specific TTS architecture, making it adaptable to different unit selection synthesizers.
Key Results
The experiments demonstrated a significant improvement in synthesis quality when using the SLMT N-best list compared to using only the original input text.
- Improved Join Cost: Figure 2 shows a box-plot comparing the "Total Join Cost" for the original text versus the best SLMT text. In the majority of cases, the synthesizer produced better results (lower join cost) when using the N-best output from the SLMT system.
- Reduced Variability: The incorporation of the MT system improved the performance in terms of the range of join cost scores and the median join cost scores.
- Robustness: In 2 out of 30 test utterances, the original text was synthesized better than any of the SLMT N-best outputs. This highlights the importance of including the original text as a candidate, especially when the MT training data is small and the N-best list might be limited.
Limitations and Future Work
- Corpus Size: The authors acknowledge that the SLMT training corpus (500 sentences) is relatively small compared to typical machine translation corpora. They suggest that a larger corpus, ideally encompassing the entire ARCTIC corpus, would likely yield even better results.
- Well-formedness of N-best Lists: It is necessary to ensure that the generated N-best list entries are grammatically correct and semantically equivalent to the input. Some entries might contain errors, and the system could potentially benefit from filtering or using only a subset of the N-best candidates.
- Alternative Parameters: While join cost and number of joins are good indicators, further investigation into other parameters for predicting synthesis quality is warranted.
- Alternative Corpora: Comparing the performance of the SLMT system when trained on the ARCTIC parallel corpus versus a different monolingual parallel corpus could provide valuable insights.
- Extending the Corpus Design: Exploring the creation of corpora with multiple source sentences for each ARCTIC target sentence could lead to a much larger and more effective training set.
- Alternative Synthesis Quality Estimation: Investigating alternative methods for estimating synthesis quality beyond join cost metrics is also a future direction.
Summary updated