Abstract
The PengYuan@PKU system addresses the "majority class bias" in Mandarin homograph disambiguation, a critical task for high-quality Text-to-Speech (TTS) systems. By automatically mining infrequent sense examples from raw text using N-gram patterns and balancing training distributions, the system effectively handles the skewed data distributions typical of natural language. In the SemEval-2010 Task 15 competition, the system achieved the top rank among nine participants, reaching a Macro Accuracy of 0.952 and a Micro Accuracy of 0.974.
Introduction
In Mandarin Chinese, homographs—words with identical written forms but different pronunciations and meanings—pose a significant challenge for automated systems. For instance, the character "背" can mean "back" (bèi) or "to recite" (bēi). Correct disambiguation is essential for natural-sounding TTS output.
The primary technical hurdle is the skewed distribution of these senses in natural language. In typical corpora, the dominant sense often accounts for over 80% of occurrences. Standard machine learning models trained on such "real-world" data are mathematically incentivized to ignore the "long tail" of infrequent senses. However, the SemEval-2010 Task 15 evaluation was specifically designed with a balanced 50/50 split between frequent and infrequent senses, causing traditional classifiers to fail. The PengYuan@PKU system was designed to overcome this bias by intentionally distorting the training distribution to be more representative of the balanced test set.
Methods
The researchers developed a supervised system based on an ensemble of seven Naïve Bayesian classifiers. Each classifier operates on a different context window size, ranging from 3 to 15 words, with the final sense determined by a majority vote. Features extracted include content words, word positions, bigrams, and Part-of-Speech (POS) tags.
N-Gram Mining
The core innovation of the system is an N-gram mining technique based on the "one sense per N-gram" heuristic. The researchers hypothesized that a specific N-gram (a sequence of 2 or 3 words) containing a homograph consistently represents the same sense in that specific context. The process involved:
- Identifying N-grams associated with infrequent senses in a small, labeled training set.
- Searching a massive, untagged corpus (People’s Daily 2001) for these specific N-grams.
- Automatically labeling these new instances as the "infrequent sense" and adding them to the training pool.
Probability Tuning
To further align the model with the balanced evaluation criteria, the team manually set the prior probability $p(S)$ to 0.5. They also optimized the additive smoothing coefficient ($\lambda$) to 0.001, which helped manage data sparseness in the augmented training sets.
Datasets and Evaluation
The system was evaluated using the SemEval-2010 Task 15 official data, which featured 16 multiple-sense target words. The "one sense per N-gram" rule was validated using the Chinese Sense-Tagged Corpus (STC), where it was found to hold true between 88.6% and 99.2% of the time, depending on the word.
The researchers found that 3-gram patterns (O+E3) were generally more effective than 2-gram patterns (O+E2). While 2-grams provided a larger volume of data, they also introduced more "noise" or incorrect labels. The 3-grams provided a cleaner, more reliable signal for the infrequent senses.
Results
The PengYuan@PKU system demonstrated superior performance, ranking 1st out of 9 participating systems in the competition. The system consistently outperformed the baseline across nearly all 16 target words, particularly for words like "背" and "车" where the initial sense distribution was highly skewed.
As shown in the results, the configuration using 3-gram mining and a small smoothing coefficient (_3.001) achieved the highest Macro Accuracy of 0.952. This confirms that the heuristic-based data augmentation successfully counteracted the natural bias toward majority classes.
Conclusion
The success of the PengYuan@PKU system highlights the importance of addressing data imbalance through targeted data mining and probability tuning. By leveraging simple but effective linguistic heuristics like "one sense per N-gram," the researchers were able to turn unlabelled data into a powerful resource for improving the recall of infrequent senses. Future work will focus on refining the identification of these senses and exploring more granular training proportions to find the absolute optimal balance for homograph disambiguation.
Summary updated