TL;DR
This paper introduces a novel Kana-Kanji translation system for Japanese text that tackles the ambiguity inherent in non-segmented input. By combining morphological, syntactic, and semantic analysis with a case grammar framework, the system achieves high translation accuracy, paving the way for more efficient Japanese text input into computers.
What Problem Does This Paper Address?
Japanese sentences are written using a mix of Kana (phonetic characters) and Kanji (ideographic characters). While Kana is used for grammatical elements and suffixes, Kanji represents the core meaning of nouns, verbs, and adjectives. This presents a significant challenge for computer input because:
- Inputting Kanji is cumbersome: There are thousands of Kanji, making direct input difficult.
- Japanese writing is not segmented: Unlike many Western languages, Japanese sentences are not typically written with spaces between words. This means a user typing in Kana must rely on a system to correctly segment the input and choose the appropriate Kanji.
- Ambiguity: The same Kana sequence can correspond to multiple Kanji words (homonyms), and the segmentation itself can be ambiguous.
The goal is to create a system that can automatically translate non-segmented Kana input into correct Kanji-Kana mixed text, mimicking how a human would write it.
Methods
The proposed system employs a two-step approach:
- Morphological Analysis and Ambiguity Extraction:
- The input Kana sentence is first analyzed morphologically.
- Ambiguous morphemes (potential Kanji words) are identified and stored in a network structure. This network represents all possible segmentation and Kanji choices.
- To manage the combinatorial explosion of possibilities, a statistical model of morpheme chains is used, focusing on characteristics like part of speech, word category (e.g., foreign origin), and length.
- A "best-first-search" heuristic is used to find a "quasi-best path" (a likely candidate morpheme string) based on a weighted morpheme number. This weighting assigns higher values to independent words and lower values to dependent words and affixes, aiming to minimize the number of "Bunsetsu" (syntactic units).
- Alternative paths near the quasi-best path are then selected to ensure more possibilities are considered.
- Syntactic and Semantic Analysis:
- This step selects the most suitable morpheme string from the generated paths.
- A case grammar framework is central to this analysis. It uses "case frames" associated with verbs to understand the relationships between words in a sentence.
- The system utilizes a detailed meaning system (a thesaurus with macro codes) and a case frame dictionary (4,600 verbs) to analyze the grammatical roles and semantic compatibility of words.
- The analysis involves matching sentence paths against case frames, considering factors like obligatory/optional case slots, postposition coincidence, meaning code coincidence, and overall case slot occupation.
- If ambiguity persists, word frequency information is used as a final tie-breaker.
Datasets
- Word Dictionary: 160,000 words.
- Case Frame Dictionary: 4,600 verbs.
- Benchmark Texts: 9 typical Japanese texts (official documents, scientific papers, legal documents) totaling about 10,000 characters for upper limit accuracy determination.
- Experimental Texts: 20,000 characters for general performance evaluation.
Key Insights and Ideas
- Unified Ambiguity Resolution: Unlike previous methods that treated segmentation and homonym ambiguity separately, this system tackles both simultaneously using syntactic and semantic analysis.
- Weighted Morpheme Number Heuristic: Modifying the "least Bunsetsu number" heuristic to a "least weighted-morpheme number" provides a more nuanced and effective way to find promising segmentation paths.
- Case Grammar for Disambiguation: The application of case grammar proves highly effective in resolving homonym ambiguity by ensuring semantic and syntactic coherence within sentence structures.
- Network Representation: Storing ambiguous morphemes in a network efficiently manages the complexity of multiple possibilities.
Key Results
The experimental system achieved a translation accuracy of 90.5%. The authors estimate that optimizing the dictionaries could improve this to about 95%.
Limitations and Future Work
- Dictionary Optimization: The authors explicitly state that optimizing the dictionaries is key to improving accuracy further.
- Context Analysis and Pragmatic Analysis: They suggest that incorporating context analysis (understanding the broader discourse) and pragmatic analysis (understanding the speaker's intent) could lead to even greater improvements in translation quality.
- Error Correction: While efforts were made to correct data errors in dictionaries for the upper limit test, the general experimental results were obtained without extensive error correction during the experiment itself.
Summary updated