Multi-Task Learning for Front-End Text Processing in TTS

Wonjune Kang, Yun Wang, Shun Zhang, Arthur Hinsvark, Qing He

TL;DR

This paper introduces a novel multi-task learning (MTL) framework for the text-to-speech (TTS) front-end, jointly optimizing text normalization (TN), part-of-speech (POS) tagging, and homograph disambiguation (HD). By leveraging shared representations and a new, balanced homograph dataset, their model achieves superior performance across all three tasks, demonstrating the power of MTL for unified TTS front-ends.

What Problem Does This Paper Address?

The front-end of a Text-to-Speech (TTS) system is crucial for converting raw text into a format that can be synthesized into speech. This process typically involves several distinct tasks, including:

  • Text Normalization (TN): Converting written text (e.g., numbers, abbreviations, dates) into their spoken equivalents (e.g., "1/2023" to "January twenty twenty-three").
  • Part-of-Speech (POS) Tagging: Assigning grammatical categories (like noun, verb, adjective) to words in a sentence.
  • Homograph Disambiguation (HD): Determining the correct pronunciation for words that are spelled the same but have different meanings and pronunciations (e.g., "lead" as a metal vs. "lead" as to guide).

Traditionally, these tasks are handled by separate modules, trained and used independently. This approach misses potential synergies: for instance, knowing a word's POS tag might help in normalizing an abbreviation, or understanding the context for HD could aid in TN. This paper argues that a multi-task learning (MTL) approach, where a single model learns to perform all these tasks simultaneously, can capture shared linguistic knowledge and improve overall performance. A key challenge in HD, specifically, is the lack of balanced datasets, which can lead to models that perform poorly on less frequent pronunciations.

Methods

The authors propose a tree-like multi-task learning architecture. The core of the model is a shared "trunk" responsible for extracting general linguistic features from the input text. This trunk operates on two parallel streams:

  1. Text Normalization (TN) Stream: This stream processes the input text after it has been tokenized for TN. It uses character-level convolutional layers, followed by a bidirectional LSTM (Bi-LSTM) and a Transformer layer to build contextualized token embeddings.
  2. Pre-trained Language Model (LM) Stream: This stream leverages a pre-trained LM (specifically, ALBERT, with its weights frozen) to extract rich contextual and lexical knowledge. The authors investigate which layers of ALBERT are most beneficial for different tasks.

These two streams are then combined using a cross-attention mechanism. The resulting fused representations are fed into separate, task-specific "heads" for TN, POS tagging, and HD. Each head consists of a feedforward network and a final classification layer tailored to its specific task.

For training, the model uses a cycle-through-tasks approach. In each training step, only one task's head is updated, using its corresponding dataset. This is necessary because the datasets for each task are distinct. The shared trunk is optimized implicitly as it contributes to all tasks. Cross-entropy loss is used for all tasks, and no explicit loss weighting is applied.

Datasets

  • Text Normalization (TN): An internal dataset of 37k training, 2k validation, and 750 test sentences.
  • Part-of-Speech (POS) Tagging: The Switchboard Dialog Act (SwDA) Corpus (125k sentences) and an internal dataset of speech assistant responses (1k sentences).
  • Homograph Disambiguation (HD):
    • Wikipedia Dataset: The dataset from Gorman et al. [8], which contains 162 American English homographs but suffers from imbalanced sentence counts per pronunciation.
  • Llama 2-Generated Homograph Dataset: A newly introduced dataset created using Llama 2-Chat 70B. This dataset aims to address the imbalance issue by providing an equal number of sentences for each pronunciation of each homograph, ensuring diversity in contexts and word senses. It contains 10 sentences per pronunciation per homograph, totaling 3,260 sentences.

Key Insights and Ideas

  • Positive Transfer is Real: The study strongly validates the intuition that jointly learning TN, POS tagging, and HD leads to positive transfer, where performance on one task benefits others. The full MTL model outperforms most single- and two-task models.
  • LM Layer Specialization: Different layers of the pre-trained ALBERT model capture distinct linguistic information. Earlier layers are more beneficial for syntactic tasks like TN and POS tagging, while later layers are better for contextual tasks like HD. This insight guides how ALBERT embeddings are incorporated.
  • Dataset Balance Matters for HD: The new Llama 2-generated dataset significantly improves HD performance, especially on its own test set, and leads to more consistent performance across different homograph pronunciations compared to using only the imbalanced Wikipedia dataset. This highlights the critical role of data quality and balance in model evaluation.
  • Unified Front-Ends are Promising: The success of this MTL approach suggests that building more unified and integrated TTS front-ends is a viable and effective direction for future research.

Key Results

  • The full multi-task model (TN + POS + HD) achieved the best overall performance, outperforming single-task and two-task models on TN and HD.
  • TN: Line Accuracy of 86.93%, WER of 2.40%.
  • POS Tagging: Accuracy of 97.18% (SwDA) and 89.91% (Internal).
  • HD: Micro-average accuracy of 96.84% and Macro-average accuracy of 93.10% on the Wikipedia dataset.
  • When trained on both the Wikipedia and the new Llama 2 HD dataset, the model achieved higher HD accuracies compared to training on Wikipedia alone.
  • On the Llama 2 test set, accuracies improved by approximately 9%.
  • The new dataset helped to reduce the performance gap between the Wikipedia and Llama 2 test sets, indicating more robust generalization.
  • Ablation studies showed that removing ALBERT significantly degraded TN and POS tagging performance, confirming the importance of the pre-trained LM. Removing the residual connection to the HD head had mixed effects, suggesting further exploration is needed for optimal LM integration.

Limitations and Future Work

  • POS Tagging Performance Drop: While the overall MTL model is superior, POS tagging performance saw a slight drop compared to a single-task POS model. The authors attribute this to POS tagging being a simpler task that might not benefit as much from the complexity of MTL, or potentially being "harmed" by other tasks as seen in prior MTL work.
  • ALBERT Integration Nuances: The impact of the residual connection from ALBERT's final layer to the HD head was inconclusive, suggesting that optimal ways to incorporate LM embeddings might require further investigation.
  • Future Work: The authors suggest that their findings can inform the development of more unified TTS front-ends. Further research could explore different MTL architectures, LM integration strategies, and potentially expand the scope of tasks included in the unified front-end. The newly created balanced HD dataset is a valuable resource for future HD research.

Summary updated

Abstract

We propose a multi-task learning (MTL) model for jointly performing three tasks that are commonly solved in a text-to-speech (TTS) front-end: text normalization (TN), part-of-speech (POS) tagging, and homograph disambiguation (HD). Our framework utilizes a tree-like structure with a trunk that learns shared representations, followed by separate task-specific heads. We further incorporate a pre-trained language model to utilize its built-in lexical and contextual knowledge, and study how to best use its embeddings so as to most effectively benefit our multi-task model. Through task-wise ablations, we show that our full model trained on all three tasks achieves the strongest overall performance compared to models trained on individual or sub-combinations of tasks, confirming the advantages of our MTL framework. Finally, we introduce a new HD dataset containing a balanced number of sentences in diverse contexts for a variety of homographs and their pronunciations. We demonstrate that incorporating this dataset into training significantly improves HD performance over only using a commonly used, but imbalanced, pre-existing dataset.
BibTeX
@inproceedings{kang2024multi,
  author = {Wonjune Kang and Yun Wang and Shun Zhang and Arthur Hinsvark and Qing He},
  title = {{Multi-Task Learning for Front-End Text Processing in TTS}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
}
Figure