TL;DR
This paper introduces a novel multi-task learning (MTL) approach to improve text-to-speech (TTS) systems by learning pronunciation from transcribed speech audio. This method effectively transfers pronunciation knowledge for words not present in the initial training data, achieving comparable performance to previous, more complex methods with a simpler implementation.
What Problem Does This Paper Address?
Modern text-to-speech (TTS) systems often rely on a "linguistic frontend" to convert text into a sequence of phonetic representations. Traditionally, this frontend is a complex pipeline involving dictionaries and rule-based systems. More recently, end-to-end sequence-to-sequence (Seq2Seq) models have emerged to simplify this, directly mapping text to pronunciation. However, these Seq2Seq models are typically trained on a fixed dataset, which limits their ability to handle words not present in that initial training data (out-of-dictionary or "extra-exclusive" words). Previous work addressed this by using transcribed speech audio, but this required training separate automatic speech recognition (ASR) models and complex pronunciation decoding. This paper aims to find a simpler, more integrated way to leverage transcribed speech audio to improve Seq2Seq frontends, specifically for these challenging extra-exclusive words.
Methods
The core of this paper is a Multi-task Learning (MTL) framework. The authors propose training a Seq2Seq TTS frontend (the "main task") simultaneously with an auxiliary task: acoustic feature regression (the "extra task").
- Main Task: The standard Seq2Seq frontend is trained to convert text sequences ($x$) into pronunciation sequences ($p$). This training uses a bootstrapped dataset ($D_{BS}$) derived from a traditional TTS pipeline.
- Extra Task: An auxiliary "acoustic decoder" is introduced. This decoder is trained to predict acoustic features ($m$) from the input text ($x$) and the pseudo pronunciation sequence ($p$) generated by the main task's frontend. This task uses a dataset of transcribed speech ($D_{TS}$), which contains the extra-exclusive words.
- Pseudo Pronunciation Generation: Since the transcribed speech dataset ($D_{TS}$) doesn't explicitly provide pronunciation sequences, a pre-trained Seq2Seq frontend is used to generate "pseudo" pronunciation sequences for the text in $D_{TS}$. This allows the MTL model to be trained with text-pronunciation-acoustic feature triplets.
- Model Architecture: The proposed architecture integrates an acoustic decoder that can attend to various intermediate representations of the main Seq2Seq frontend (e.g., encoded text, attention hidden states, context vectors, decoder hidden states). This allows flexibility in how the acoustic information is leveraged.
The key novelty lies in jointly training these two tasks, allowing the acoustic regression task to act as an "inductive bias" to improve the main TTS frontend's ability to generalize to the extra-exclusive words present in the transcribed speech data.
Datasets
- Bootstrapping Dataset ($D_{BS}$): Derived from normalized text subsets of LibriSpeech [32]. This dataset contains approximately 206k sentences and is used for training the main Seq2Seq TTS frontend. It excludes sentences with out-of-dictionary words to ensure high-quality initial pronunciation targets.
- Transcribed Speech Dataset ($D_{TS}$): A collection of text-audio pairs from three speakers of the Hi-Fi TTS dataset [34], totaling 103k sentences (81.7 hours). This dataset is crucial as it contains the "extra-exclusive" words that the baseline model struggles with.
- Evaluation Datasets:
- Extra-exclusive word tokens: A test set derived from $D_{TS}$ by generating reference pronunciations using Festival. This set contains 1.6k extra-exclusive word tokens.
- Out-of-vocabulary (OOV) word tokens: A test set created from Hi-Fi TTS sentences containing OOV words (not in the baseline's dictionary or $D_{BS}$), processed through Festival. This set includes 4.4k sentences.
Key Insights and Ideas
- MTL as a Simpler Alternative: The paper demonstrates that MTL can achieve similar improvements to more complex methods (like the FA-based approach in [8]) for acquiring pronunciation knowledge from transcribed speech, but with a significantly simpler implementation flow.
- Acoustic Regression as an Inductive Bias: The acoustic feature regression task effectively guides the Seq2Seq frontend to learn better representations, particularly for words it hasn't seen before. This is because the acoustic signal inherently contains pronunciation information, even for novel words.
- Importance of Intermediate Representations: The flexibility to attend to different layers of the Seq2Seq frontend allows the acoustic decoder to extract relevant information at various stages of processing, contributing to the overall effectiveness.
- Generalization to Extra-Exclusive Words: The study highlights that MTL is particularly effective at improving generalization to "extra-exclusive" word types, which are words present in the auxiliary task's data but not in the main task's initial training data.
Key Results
- Extra-Exclusive Words: The proposed MTL-based method achieved a 3.7% absolute improvement in overall accuracy (WAcc) for extra-exclusive word tokens compared to the baseline Seq2Seq frontend. This performance was comparable to the FA-based method from prior work.
- Main-Covered Words: For words already present in the bootstrapping dataset, all methods (baseline, FA, and MTL) performed very well, with WAcc > 99.9%.
- OOV Words: The MTL method showed comparable performance to the baseline for OOV words, though it slightly underperformed in terms of PER and WAccP. The authors acknowledge this limitation.
- Acoustic Feature Choice: Mel-spectrograms were found to be superior to MFCCs as acoustic features for the extra task.
- Loss Function: For the acoustic loss ($L_a$), an L1 loss was found to be more effective than L2 loss or a multimodal LMM+NLL loss.
Limitations and Future Work
- OOV Word Performance: The paper acknowledges that the MTL method did not significantly improve performance on out-of-vocabulary (OOV) words and even slightly underperformed the baseline in some metrics.
- Future Work: The authors explicitly state they will work on improving the generalization to OOV words in future research. They also suggest that the proposed MTL-improved Seq2Seq frontends could be used to expand dictionaries by acquiring pronunciations for extra-exclusive words.
Summary updated