Development of Basic Practical Techniques for Japanese Letter String Processing - Automatic Keyword Extraction and Automatic Reading

K. Araki, K. Hinatsu, K. Itayama, T. Sahara, Y. Sakagami, F. TakanoCOLING 1982

TL;DR

This paper introduces a novel system for processing Japanese text, tackling the inherent complexities of its mixed character set and multiple readings. The authors developed efficient algorithms for automatic keyword extraction and character reading, achieving high precision and enabling Japanese text processing to rival that of Western languages.

What Problem Does This Paper Address?

Japanese presents a unique challenge for natural language processing due to its writing system, which combines Chinese characters (kanji) with phonetic scripts (kana) without spaces. Kanji characters often have multiple readings that depend heavily on context, making it difficult to automatically segment text, extract meaningful keywords, and determine the correct pronunciation. This complexity historically required extensive manual effort for tasks like indexing and terminology extraction, necessitating large, specialized keyboards for input. The authors aimed to automate these processes, making Japanese text processing more efficient and comparable to Western languages.

Methods

The core of the proposed solution lies in a two-subsystem approach:

  • K-KACS (Kanji-Kana Automatic Conversion System): This system focuses on automatically determining the correct reading of kanji characters within a given context. It leverages a comprehensive dictionary of special readings (approximately 25,000 for 2,000 characters) and an efficient algorithm to select the appropriate reading with high precision.
  • JAKAS (Japanese Keyword Automatic Selection): This subsystem handles automatic keyword extraction. It employs a strategy that first identifies terms by matching letter strings against a dictionary of approximately 8,000 words, particularly those where kanji are integral parts of words rather than just prefixes or suffixes. For longer strings lacking such special characters, it uses dictionary terms to segment sentences.

The authors also developed specialized dictionaries, including one for characters that function as affixes or prepositions (around 500) and another for terms where characters are key components (around 8,000 words).

Datasets

The paper does not explicitly detail specific training or evaluation datasets in the traditional sense. Instead, it describes the creation of custom dictionaries:

  • A dictionary of approximately 500 characters that function as affixes or prepositions.
  • A dictionary of approximately 8,000 words where kanji are important components.
  • A collection of approximately 25,000 special readings for about 2,000 Chinese characters.

These dictionaries, along with the developed algorithms, form the basis of their system's performance.

Key Insights and Ideas

  • Context is King for Kanji Readings: The paper highlights that the multi-faceted nature of kanji readings is not arbitrary but highly context-dependent, necessitating sophisticated algorithms to disambiguate.
  • Hybrid Approach for Keyword Extraction: The combination of dictionary-based matching for known terms and contextual segmentation for longer strings provides a robust method for extracting keywords from Japanese text.
  • Efficiency Through Simplicity: The authors emphasize that despite the complexity of the problem, their system's dictionary is relatively small and its logic is simple, leading to easy implementation, maintenance, and high processing speed.
  • Bridging the Gap: The development of these techniques effectively closes the gap in processing efficiency between Japanese and Western languages, a significant achievement.

Key Results

The system achieves a high precision rate of over 99.94% for selecting the correct reading of Chinese characters. The authors state that their system has been adopted by JICST for its information file production and services, processing over 400,000 citations annually, leading to cost savings. The development has made Japanese text processing capabilities comparable to those of Western languages.

Limitations and Future Work

The paper does not explicitly detail limitations or suggest future work. However, the focus on "basic practical techniques" implies that further refinements and expansions of the system are possible. The success in achieving high precision with a relatively simple logic suggests that further optimization or integration with other NLP techniques could yield even greater benefits.

Summary updated

BibTeX
@inproceedings{araki1982development,
  author = {K. Araki and K. Hinatsu and K. Itayama and T. Sahara and Y. Sakagami and F. Takano},
  title = {{Development of Basic Practical Techniques for Japanese Letter String Processing - Automatic Keyword Extraction and Automatic Reading}},
  booktitle = {International Conference on Computational Linguistics},
  year = {1982},
}
Figure