TL;DR
This paper introduces DLM, a novel decoupled learning model designed to tackle the "long-tail" problem in Mandarin polyphone disambiguation for text-to-speech systems. By separating representation and classification learning and incorporating novel attention mechanisms for context transition, DLM significantly boosts accuracy, especially for rare polyphonic characters.
What Problem Does This Paper Address?
Chinese, unlike many other languages, has a significant number of polyphonic characters – characters that share the same written form but have multiple pronunciations. Accurately determining the correct pronunciation for these characters based on context is crucial for natural-sounding text-to-speech (TTS) systems. However, existing datasets for this task suffer from a "long-tail" distribution: a few common polyphones dominate the dataset, while many rare polyphones have very few examples. This imbalance makes it difficult for models to learn effectively, particularly for the underrepresented rare characters. Furthermore, previous models often relied on context from only one dimension (either local or global), limiting their ability to capture nuanced linguistic information.
Methods
The core innovation of DLM lies in its decoupled learning approach. Instead of training representation and classification simultaneously, DLM splits this into two stages:
- Representation Learning: The model is trained on the original, imbalanced dataset using a uniform sampler. This stage focuses on learning robust feature representations from the data as it is.
- Classification Learning: The model's classifier is then retrained using a "reverse sampler" on the same imbalanced dataset. This sampler over-weights the tail classes, allowing the classifier to better learn from the rare examples.
DLM also introduces two improved attention mechanisms to better capture context:
- Convolution Attention: This mechanism focuses on extracting local features within a sliding window, similar to how Convolutional Neural Networks (CNNs) operate.
- Linear Attention: This mechanism, combined with BiLSTM, focuses on global features across the entire sequence. It's designed to be computationally more efficient than traditional softmax attention.
Crucially, DLM integrates these two attention mechanisms with a gradual conversion strategy. This strategy allows the model to transition its focus from local context to global context during training, mimicking how humans process information.
Datasets
- CPP Dataset: This is the primary benchmark dataset used for training and evaluation. It's noted to be imbalanced and contains 99,264 sentences with 623 different polyphonic characters.
- BCP Dataset: A new, balanced dataset created by the authors using In-Context Learning (ICL) with large language models (LLMs). This dataset was used to verify the effectiveness of decoupled learning and to evaluate the best sampler for representation learning.
Key Insights and Ideas
- Decoupled Learning is Key for Long-Tail: The paper strongly suggests that separating representation learning (on original data) and classification learning (on re-balanced/reversed data) is more effective for long-tail problems than unified training.
- Original Distribution Aids Representation: The initial stage of training on the imbalanced dataset is crucial for learning good representations, as it reflects the true data distribution.
- Reverse Sampler Boosts Classification: The reverse sampler in the second stage is vital for improving the classifier's performance on tail classes.
- Local-to-Global Context Transition Matters: Simply focusing on local or global context isn't enough. A gradual transition between the two, facilitated by the proposed attention mechanisms and conversion strategy, leads to better performance.
- ICL for Data Generation: The paper demonstrates a practical and effective method for generating balanced datasets using LLMs and in-context learning, which can be a valuable tool for future research.
Key Results
- DLM achieved an overall accuracy of 99.07% on the CPP dataset, outperforming several strong baseline models.
- For long-tailed characters, DLM demonstrated significant improvements, with many achieving 100% accuracy.
- The decoupled learning approach was shown to be superior to unified training methods.
- The gradual conversion strategy (specifically cosine decay) outperformed fixed strategies (equal weight).
- DropConnect was found to be more effective than standard Dropout for regularization.
- The optimal kernel size for convolution attention was found to be 5.
Limitations and Future Work
- Human Intervention in Corpus Generation: While ICL is powerful, some cases of word segmentation and polyphonic character labeling still require manual intervention.
- ROBERTa's Inference Speed: The reliance on RoBERTa for initial feature extraction, while effective, leads to relatively large model parameters and slower inference speeds, which is a concern for real-time TTS systems.
- Label Granularity: The paper suggests that labels should ideally be character-pronunciation pairs, not just pronunciations, to better capture nuances.
- Future Work: The authors aim to optimize the corpus creation approach to minimize human intervention, reduce model parameters, and improve inference speed. They also plan to build character-pronunciation pair labels for future tasks.
Summary updated