An Automatic Processing of the Natural Language in the Word Count System

Hiroshi Nakano, Shin’ichi Tsuchiya, Akio TsuruokaCOLING 1980

TL;DR

This paper presents a compact, fast, and accurate (90%+) automatic processing system for Japanese natural language. It tackles the challenge of Japanese text processing, which is complicated by its mixed writing systems and lack of spaces between words, by developing novel methods for segmentation, transliteration, and part-of-speech tagging.

What Problem Does This Paper Address?

Processing natural language data, especially for tasks like word counting, requires clean and structured input. For Japanese, this presents a significant challenge due to its complex writing system, which combines Kanji (Chinese characters), Hiragana and Katakana (syllabaries), Roman letters, and numerals, often without spaces between words. This ambiguity makes automatic segmentation (breaking text into words) and transliteration (converting Kanji to phonetic representations) difficult. Traditional word count systems often offload these complex tasks to humans, who are prone to errors. The authors aim to automate these processes to create a more efficient and reliable system.

Methods

The core of the system is a three-pronged approach to automatic natural language processing:

  • Segmentation: This is a crucial step, as Japanese lacks explicit word delimiters. The system uses a combination of techniques:
  • A character combination table (Table 3) that identifies high-ratio segmental points based on character types (Kanji, Hiragana, Katakana, Alphabet, Numeral, Sign).
  • A special rule for Hiragana-Hiragana combinations, treating the Hiragana "が" as a delimiter due to its frequent use as a particle.
  • A table of 300 records (Figure 5) for specific character strings that cannot be segmented by the general rules, providing word lengths, parts of speech, and conjugations.
  • Transliteration from Kanji to Kana/Roman Letters: This process involves determining the correct phonetic reading for Kanji characters.
  • Kanji are categorized into groups based on their reading complexity.
  • A context-dependent approach is used, analyzing characters before and after a Kanji to select the most appropriate reading from a small table. This table uses an "operation code" to guide the selection based on character combinations.
  • Classification of Parts of Speech: The system employs three methods:
  • Utilizing information from the segmentation table (Figure 5).
  • Rule-based classification based on the last character of a word (e.g., if the last character is Kanji, Katakana, or alphabet, it's likely a noun).
  • Word combination analysis, particularly for nouns and verbs with particles or auxiliary verbs, using a formulaic approach (Figure 7).

A "supervisor" program then checks and corrects the results from these three processes, leveraging information from each stage to improve overall accuracy.

Datasets

The paper mentions using a dictionary compiled by the National Language Research Institute, containing approximately 5 million words. This dictionary includes word frequencies, parts of speech, word origin classes, and thesaurus numbers. The thesaurus itself, "Bunrui goihyo," contains about 38,000 words. For evaluation, they used a sample of 2500 words from a high school textbook.

Key Insights and Ideas

  • Leveraging Ambiguity as a Feature: The authors cleverly turn the complexity of Japanese writing systems (mixed scripts, no spaces) from a demerit into a merit, using these very features to aid in processing.
  • Compactness and Speed: The system is designed to be compact, utilizing a small table (300 units for segmentation) and achieving high speed.
  • Context is King for Transliteration: The success of Kanji transliteration hinges on analyzing the surrounding characters, demonstrating the importance of local context in linguistic processing.
  • Multi-pronged POS Tagging: Combining table lookups, form-based rules, and word combination analysis provides a robust approach to part-of-speech tagging.
  • Supervisor for Error Correction: The supervisor program is a critical component, acting as a post-processing step to catch and correct errors, significantly boosting overall accuracy.

Key Results

The system achieved approximately 90% correct answers in experiments. Specifically, the paper notes the following accuracy rates:

  • Segmentation: 91.3%
  • Transliteration from Kanji to Kana: 95.7%
  • Classification of parts of speech: 97.0%

The system is described as "very fast" and its output can be generated on various devices, including line printers and displays.

Limitations and Future Work

The authors acknowledge that the automatic processing program currently makes more mistakes than humans. However, they can predict where these mistakes will occur, allowing for easier correction. They also note that the system does not completely cover all cases, particularly phenomena like "rendaku" or "renjo" (sound changes in compound words), for which there are no clear rules.

Future work could involve improving the supervisor program and the underlying tables. They suggest that achieving 100% accuracy might be challenging due to the inherent non-standardization of the Japanese writing system. Furthermore, a complete program would require processing based on syntax and meaning, which is beyond the current scope of their efforts.

Summary updated

BibTeX
@inproceedings{nakano1980automatic,
  author = {Hiroshi Nakano and Shin’ichi Tsuchiya and Akio Tsuruoka},
  title = {{An Automatic Processing of the Natural Language in the Word Count System}},
  booktitle = {International Conference on Computational Linguistics},
  year = {1980},
}
Figure