TL;DR
This paper introduces a novel Multi-Task Learning (MTL) approach for predicting prosodic boundaries in Mandarin speech. By jointly training models for Prosodic Word (PW), Prosodic Phrase (PPH), and Intonational Phrase (IPH) prediction, they achieve improved performance and a significantly smaller model size compared to independent models.
What Problem Does This Paper Address?
Mandarin, unlike English, is a tonal language with a complex prosodic structure that plays a crucial role in speech synthesis naturalness and intelligibility. This structure is typically hierarchical, consisting of Prosodic Words (PWs), Prosodic Phrases (PPHs), and Intonational Phrases (IPHs). Previous research often tackles the prediction of boundaries for these units as separate, independent tasks. However, this approach overlooks the inherent dependencies between these prosodic levels, potentially limiting overall prediction accuracy. This paper aims to bridge this gap by proposing a unified model that leverages these dependencies through Multi-Task Learning.
Methods
The core of the proposed method is a Multi-Task Learning (MTL) architecture built upon a Bidirectional Long-Short Term Memory (BLSTM) and Conditional Random Field (CRF) backbone. The MTL framework treats the prediction of PW, PPH, and IPH boundaries as three distinct subtasks.
The model begins with a feature sharing module that processes character-level features (Chinese Character, Chinese Word Segmentation, Part of Speech, word length, and distance) through embedding layers and a Multi-layer Feedforward Neural Network (MFNN) to create a unified feature representation.
The task-specific module then utilizes this shared representation. Crucially, the subtasks are designed to depend on each other:
- The PW subtask is independent and takes the shared feature embedding as input.
- The PPH subtask takes both the shared feature embedding and the prediction output of the PW subtask as input, fused by a Feedforward Neural Network (FNN).
- The IPH subtask similarly takes the shared feature embedding and the prediction outputs of both PW and PPH subtasks as input.
This hierarchical dependency structure allows the model to learn from the relationships between different prosodic levels. The overall loss function is a weighted sum of the individual subtask losses, enabling joint optimization.
Datasets
The paper utilizes a proprietary dataset from Databaker, which contains approximately 150,000 Mandarin sentences. This dataset was labeled by two linguistic experts and has been double-checked for consistency. The data is split into a training set (90%) and a test set (10%). The paper does not introduce a new benchmark dataset but rather uses its internal, expert-labeled data.
Key Insights and Ideas
- Leveraging Inter-Task Dependencies: The most significant insight is that prosodic boundary prediction tasks (PW, PPH, IPH) are not truly independent. By modeling these dependencies (e.g., PPH depends on PW), the MTL approach can achieve better overall performance.
- Model Compression: The MTL architecture leads to a substantial reduction in model size (approximately 55% smaller) compared to training individual BLSTM-CRF models for each task. This is a critical advantage for practical deployment in speech synthesis systems.
- Unified Optimization: Treating prosodic boundary prediction as a single, multi-task problem allows for a more holistic optimization of the model, potentially leading to more robust predictions across all levels.
Key Results
- The proposed MTL model achieved an overall improvement of 0.8% in prediction performance compared to benchmark models.
- Specifically, the F1 score for PPH and IPH tasks saw improvements of 0.49% and 0.35%, respectively, while the PW task had a slight decrease of 0.02%.
- The model size was significantly reduced, with the MTL model being approximately 45% the size of the benchmark models.
Limitations and Future Work
- Loss Weight Tuning: The paper notes that the optimal weighting of subtask losses in the MTL framework is crucial. While they experimented with a ratio of 0.5:1.0:2.0 (PW:PPH:IPH), the results were not as expected, suggesting that further research into optimizing these weights is needed.
- Pre-training Word Vectors: The authors plan to explore incorporating pre-trained word vectors as input to further enhance feature representation.
- Optimizing Loss Weighting Methods: They intend to investigate more advanced methods for adjusting loss weights, potentially drawing from multi-objective optimization techniques.
Summary updated