TL;DR
This paper tackles the challenge of correctly pronouncing Japanese proper names that have multiple possible readings. Instead of relying on expensive, manually curated dictionaries, the authors propose a novel method that leverages the vastness of the Web to automatically build training data and a machine learning classifier to disambiguate pronunciations with impressive accuracy.
What Problem Does This Paper Address?
Text-to-speech (TTS) systems struggle with words that are spelled the same but have different pronunciations (heteronyms). While previous research focused on common words by using part-of-speech or sense tags, these methods fail for proper names. Proper names, especially those for locations, people, and organizations, are a significant source of heteronyms in Japanese. For instance, a single Kanji string can represent multiple distinct pronunciations. The core problem is how to automatically determine the correct pronunciation for these ambiguous proper names in a way that is scalable and doesn't require laborious manual annotation.
Methods
The paper proposes a two-pronged approach:
- Basic Method (Web Co-occurrence): This method leverages the observation that proper names written in Kanji often appear alongside their pronunciations in Hiragana or Katakana on the same Web page. By searching the Web for a given Kanji name and its potential readings, the authors count the co-occurrence frequency. The most frequent co-occurring reading is then selected as the most likely pronunciation. This is a simple yet effective heuristic.
- Improved Method (Machine Learning Classifier): This method builds upon the basic approach by using the Web to automatically create a training corpus. For a given ambiguous word (W) and its set of possible readings {Rk}, the system queries the Web for "W AND Rk" and collects snippets. These snippets, along with the corresponding reading Rk, form the training data. A machine learning classifier (specifically, a decision tree is mentioned) is then trained on this data. The features used for classification include the part-of-speech of surrounding words and keywords within the snippet, effectively grounding the disambiguation task in real-world context. The "window size" (M) refers to the number of surrounding words considered as features.
Datasets
- Japan Post Postal Address Lists: Used to extract a large list of Japanese location names and their associated pronunciations. This formed the basis for identifying ambiguous names.
- EDR Corpus: A corpus of Japanese newspaper sentences, where each word is tagged with its part-of-speech and pronunciation. This was used to extract sentences containing location heteronyms for testing.
The paper effectively uses the Web as a dynamic, large-scale source for building training data, rather than relying on pre-existing, static corpora for this specific task.
Key Insights and Ideas
- The Web as a Scalable Training Data Source: The most significant insight is the successful use of the Web to automatically generate a pronunciation-tagged corpus. This bypasses the need for expensive, manual annotation, making the approach highly scalable.
- Co-occurrence as a Strong Signal: Even the basic method, relying solely on Web co-occurrence frequency, achieves surprisingly high accuracy, suggesting that common pronunciations are indeed dominant online.
- Contextual Features are Crucial for Ambiguity: The improved method, incorporating contextual features (part-of-speech and keywords), significantly outperforms the basic method, demonstrating the importance of surrounding text for disambiguation.
- Proper Names are a Major Source of Heteronyms: The paper highlights that proper names, particularly locations, are a dominant source of pronunciation ambiguity in Japanese.
Key Results
- Basic Method Accuracy: Achieved an average accuracy of 89.2% for Hiragana and 86.6% for Katakana on open data.
- Improved Method Accuracy: Outperformed the basic method, achieving up to 92.9% accuracy for Hiragana and 89.9% for Katakana with a window size of 10.
- High Coverage: The Web-based approach demonstrated near 100% coverage for the identified ambiguous name pairs.
- Ambiguity Impact: While the improved method performed well across the board, accuracy decreased for highly ambiguous cases (average ambiguity of 7.1), though it still significantly outperformed the basic method.
Limitations and Future Work
- Dependence on Web Co-occurrence: The methods rely on the target word and its pronunciation appearing together on Web pages. If such co-occurrences are rare or non-existent, the method may not work.
- Limited Training Data Size (L): The paper notes that the limitation of the training data size (L) in Section 2.3 can sometimes lead to lower accuracy compared to the basic method's Web frequency, especially when the Web distribution is very sharp.
- Future Work: The paper suggests that similar contextual approaches could be applied to transliteration problems between different languages, where multiple translation candidates exist.
Summary updated