Pronunciation Ambiguities in Japanese Kanji

Wen ZhangACL 2023

TL;DR

This paper tackles the challenge of Japanese kanji homograph disambiguation for text-to-speech (TTS) systems. The authors introduce the first publicly available annotated dataset of single kanji pronunciations and demonstrate that a logistic regression model significantly outperforms a simple baseline in predicting the correct pronunciation based on context.

What Problem Does This Paper Address?

Japanese writing is a complex system that combines Chinese characters (kanji) with two native phonetic scripts (hiragana and katakana). A significant challenge for natural-sounding TTS systems is that many kanji characters have multiple pronunciations, often derived from different historical borrowings from Chinese or from native Japanese words. This phenomenon, known as homograph ambiguity, means that the same written kanji can be pronounced differently depending on its context and intended meaning. For instance, the kanji 山 can be read as "yama" (native Japanese for mountain) or "san" (Sino-Japanese). Accurately predicting the correct pronunciation is crucial for TTS intelligibility and naturalness. While homograph disambiguation has been studied in other languages, research in Japanese, particularly for single kanji characters, has been limited due to a lack of comprehensive annotated data.

Methods

The core of this research lies in creating a new dataset and applying a machine learning approach. The authors developed a logistic regression (LR) classifier to predict the appropriate pronunciation of a kanji character. This model utilizes n-gram features, which include tokens (words or characters) immediately preceding and following the target kanji, as well as bigrams and skip-grams that capture local context. They also established a baseline model that simply predicts the most frequent pronunciation for each kanji, serving as a benchmark for the LR classifier's performance. The LR classifier was trained per-kanji, meaning a separate model was built for each of the kanji homographs studied.

Datasets

The primary dataset used and introduced in this paper is a newly annotated Japanese single kanji character pronunciation dataset. This dataset was constructed by combining information from:

  • Jiten: An online Japanese dictionary containing kanji readings.
  • Universal Dependencies (UD) Japanese-GSD: A corpus of Japanese sentences annotated with grammatical information, used to provide contextual information for kanji characters.

The authors extracted 26 unique kanji homographs from this combined resource. They also mention releasing two parts of their data:

  1. A tab-separated values (TSV) file of 2,136 commonly used kanji homographs with their readings and reading types.
  2. An annotated UD Japanese-GSD dataset with kanji homograph readings in context.

Key Insights and Ideas

  • The "per-kanji" approach is effective: Training separate models for each kanji homograph, rather than a single global model, allows the system to learn specific contextual patterns for each ambiguous character.
  • Context is king (and n-grams help): The LR classifier's ability to leverage surrounding words (n-gram features) is key to its success in disambiguating kanji pronunciations, significantly outperforming a baseline that ignores context.
  • Part-of-Speech (POS) tagging is not a silver bullet for Japanese kanji: The authors found that POS tags are not reliable features for Japanese kanji disambiguation because many kanji have the same POS regardless of their pronunciation, and some kanji don't have a clearly definable POS.
  • Data scarcity is a major hurdle: The limited availability of annotated data has historically hindered research in this area, highlighting the importance of their released dataset.
  • Subtle linguistic features matter: The paper touches upon the influence of formality and geographical features on kanji pronunciation, suggesting these are important factors for robust disambiguation.

Key Results

The research demonstrates a significant improvement in accuracy using the logistic regression classifier compared to the baseline.

  • The baseline model achieved an average micro and macro accuracy of .67.
  • The logistic regression classifier achieved an average micro and macro accuracy of .83.
  • On a per-kanji basis, LR accuracies ranged from 0.57 to 1.00, significantly outperforming baseline accuracies which ranged from 0.43 to 0.86.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Human error in annotation: As the data was annotated by a single author, there's a possibility of human error, which could be mitigated by peer review and publication of an amended version.
  • Limited corpus: The UD Japanese-GSD is one of many corpora, and using others could expand the dataset.
  • Single kanji focus: The research primarily focuses on single kanji homographs. Future work could expand to kanji combinations and phrases, which introduces further complexities in word segmentation and pronunciation selection.
  • Feature engineering: While n-gram features were used, other features could potentially improve model performance.
  • Comparison with existing tools: A comparison with systems like MeCab's embedded kanji disambiguation is suggested for future work.
  • Formality and geographical features: The LR classifier struggled with formality distinctions, indicating a need for more robust handling of these linguistic nuances.
  • Imbalanced pronunciation classes: The model had difficulty with kanji having highly imbalanced pronunciation class sizes, suggesting further investigation into handling such imbalances.

Summary updated

BibTeX
@inproceedings{zhang2023pronunciation,
  author = {Wen Zhang},
  title = {{Pronunciation Ambiguities in Japanese Kanji}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure