Paraphrase generation to improve text-to-speech synthesis

Ghislain Putois, Jonathan Chevelu, Cédric BoidinInterspeech 2010

TL;DR

This paper explores using paraphrase generation to improve the naturalness of Text-to-Speech (TTS) systems. By automatically rephrasing sentences before synthesis, they aim to leverage diverse wording to enhance the perceived quality of the synthesized speech, especially in scenarios where a single, fixed voice corpus might lead to repetitive or unnatural output.

What Problem Does This Paper Address?

Modern Text-to-Speech (TTS) systems, particularly those using unit selection (concatenating pre-recorded speech segments), achieve high quality when the voice corpus is tailored to a specific task and speaking style. However, creating such specialized corpora for every possible application is impractical. This leads to a challenge: how to improve the perceived quality of synthesized speech when the available voice corpus, though high-quality for its domain, might not offer the optimal wording for every message. The authors propose that by paraphrasing the input text, they can generate alternative phrasings that might better suit the available speech units, thereby improving the overall naturalness and quality of the synthesized output.

Methods

The proposed system integrates a paraphrase generator with a unit selection TTS system.

  • Paraphrase Generator: This component is built using a statistical machine translation (SMT) approach. It leverages a French-to-French corpus trained on a bilingual aligned corpus (French-English, using English as a pivot language). This method allows for the creation of large paraphrase tables, enabling the generation of paraphrases for a wide range of sentences. Several heuristics are used to prune the paraphrase table, and specific weighting factors are empirically set for the MOSES scoring function.
  • TTS Selector: This component ranks the generated paraphrases based on the speech synthesizer's score. The rationale is that the speech synthesizer's score is often correlated with subjective perception of paraphrases. The system selects the "best" paraphrase based on this score. To balance paraphrase variability and correctness, the paraphrase generator is constrained to output only a 10-best set of paraphrases.
  • TTS Synthesizer: A standard unit selection concatenative TTS system (Baratinoo) is used, which minimizes a scoring function based on concatenation and target costs to vocalize the selected paraphrase.

Datasets

  • Paraphrase Generation Training: The Europarl corpus, a large, multi-lingual, freely available aligned corpus of European parliament debates, was used. Specifically, the French-English pair was utilized, with French as the target language for paraphrasing and English as the pivot language.
  • TTS System: A female voice corpus, consisting of approximately 5 hours of segmented recorded speech, was used for the TTS synthesizer.
  • Evaluation: 100 French sentences were randomly extracted from the Europarl corpus to serve as a test set for the system.

Key Insights and Ideas

  • Paraphrasing for TTS is a viable concept: The core idea is that by offering alternative phrasings, the TTS system can potentially find better matches within its voice corpus, leading to more natural-sounding speech.
  • SMT techniques are adaptable for paraphrase generation: The paper demonstrates that statistical machine translation frameworks, typically used for cross-lingual translation, can be effectively adapted for monolingual paraphrase generation by using a pivot language.
  • The importance of joint evaluation: Evaluating such a hybrid system requires considering both the linguistic quality of the paraphrases (do they mean the same thing?) and the acoustic quality of the synthesized speech.
  • Subjective evaluation is crucial: Objective metrics alone are insufficient. The paper highlights the necessity of human evaluation (both linguistic and acoustic) to truly assess the system's effectiveness.
  • Challenges in achieving significant MOS improvement: Despite the potential of paraphrasing, achieving a statistically significant improvement in Mean Opinion Scores (MOS) for acoustic quality proved difficult, suggesting that the TTS system's existing capabilities might already be sufficient for the tested corpus.

Key Results

  • Paraphrase Generator Performance (Standalone): Only 22 out of 100 paraphrases were rated as correct by both judges in terms of syntactic and semantic evaluations. The kappa statistic for agreement was 0.69 ("Substantial").
  • Complete System Performance (Linguistic): When considering the entire system, linguistic performance improved to 30 correct paraphrases. This increase is partly attributed to allowing the source sentence itself to be considered a valid paraphrase if no better option was found. The kappa statistic rose to 0.76 ("Substantial").
  • Complete System Performance (Acoustic): The subjective MOS evaluation showed a slight improvement of 0.01 points for the complete system compared to the standalone TTS system, but this difference was not statistically significant within a 95% confidence interval.
  • Acoustic Score Decrease: The 1-best sentences output by the TTS selector obtained lower acoustic scores compared to the original sentences. This decrease was 21% ± 29% when considering all 100 sentences and 15% ± 20% when considering only the 30 linguistically correct paraphrases.

Limitations and Future Work

  • Limited Linguistic Correctness: The paraphrase generator only produced correct paraphrases for 30% of the sentences, indicating it's not yet ready for fully automatic service and might be better suited to assist human prompt designers.
  • Lack of Significant Acoustic Improvement: The primary goal of improving acoustic quality via paraphrasing was not met with a statistically significant MOS improvement.
  • Corpus Dependency: The authors suggest that the tested corpus might have been too well-adapted to the task, providing sufficient acoustic unit coverage, thus masking the benefits of paraphrasing. Future work should explore systems with more limited voices and different task categories.
  • Paraphrase Variability: The mean Character Error Rate (CER) between source and paraphrase was low (0.18 ± 0.13), indicating limited variability in the generated paraphrases.
  • Non-correlation of Components: The paraphrase generator and TTS selector are currently uncorrelated. An acoustic disfluency in the source sentence is likely to persist across paraphrases.
  • Future Directions:
    • Adapt linguistic evaluation to the vocal modality.
  • Experiment on more limited voices and different task categories.
  • Explore joint optimization of linguistic and acoustic scores using a more permissive decoder and a global scoring function, potentially using methods proposed in reference [7].

Summary updated

BibTeX
@inproceedings{putois2010paraphrase,
  author = {Ghislain Putois and Jonathan Chevelu and Cédric Boidin},
  title = {{Paraphrase generation to improve text-to-speech synthesis}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure