A Probabilistic Approach to Compound Noun Indexing in Korean Texts

Hyouk R. Park, Young S. Han, Kang H. LeeCOLING 1996

TL;DR

This paper tackles the challenge of accurately indexing Korean documents by effectively handling compound nouns. They propose a probabilistic method that decomposes compound nouns and evaluates their component parts based on their distribution across a document set, achieving improved recall without sacrificing precision.

What Problem Does This Paper Address?

Automatic indexing is crucial for making large document collections searchable and understandable. A key aspect of this is selecting the right "index terms" that represent the document's content. In Korean, compound nouns (words formed by combining multiple nouns) pose a significant challenge. These compound nouns often carry specific meanings, but their segmentation can be ambiguous due to flexible spacing rules. Simply treating each component of a compound noun as an index term can boost recall (finding more relevant documents) but often degrades precision (returning irrelevant documents). Conversely, using only the full compound noun as an index term might improve precision but miss relevant documents that don't use the exact same compound. This paper aims to bridge this gap by finding a way to decompose compound nouns and select the most informative component nouns as index terms, thereby improving both recall and precision.

Methods

The core of the proposed method lies in a probabilistic approach to evaluating the potential of component nouns derived from compound nouns.

  1. Dictionary Building: A noun dictionary is automatically constructed from the document set itself. This reduces ambiguity by focusing only on nouns present in the corpus. Non-nominal words (verbs, adverbs, etc.) are identified and removed.
  2. Tokenizing and Compound Noun Analysis: The system first identifies simple and compound nouns using the built dictionary and stemming rules. For compound nouns, it explores all possible decompositions into simpler nouns.
  3. Probabilistic Evaluation: The key innovation is evaluating each component noun's "meaning" by its distribution across the entire document set. This distribution is represented as a probability vector. The similarity (or dissimilarity) between the distribution of a component noun and the distribution of the document set it belongs to is measured using information-theoretic divergence.
  4. Index Term Selection: For each compound noun, the decomposition that yields the lowest divergence (meaning the component nouns are most consistent with the document's overall content distribution) is selected.
  5. Index Weighting: Finally, standard inverse document frequency (IDF) is used to weight the selected index terms.

Datasets

The experiments were conducted on a dataset of 1,000 Korean science abstracts. The paper mentions that this dataset was used to automatically construct the noun dictionary and for evaluating the proposed method. No new benchmark datasets were introduced.

Key Insights and Ideas

  • Document-Specific Dictionaries: Building the noun dictionary directly from the corpus significantly reduces ambiguity compared to using a general, pre-defined dictionary.
  • Distribution as Meaning: Representing a term's "meaning" by its probabilistic distribution across a document set is a powerful way to capture its contextual relevance.
  • Information-Theoretic Divergence for Evaluation: Using information-theoretic divergence to measure the similarity between term distributions provides a robust metric for evaluating the quality of compound noun decompositions.
  • Balancing Recall and Precision: The probabilistic evaluation framework allows for the selection of component nouns that improve recall without unduly sacrificing precision, a common trade-off in indexing.

Key Results

Experiments showed that the proposed probabilistic approach achieved a 33% increase in retrieval performance compared to indexing methods that did not analyze compound nouns. Furthermore, the performance was comparable to manual decomposition by human experts. The method was also more efficient than the bigram method in terms of the number of index terms and the average number of retrieved documents per query.

Limitations and Future Work

  • Ambiguity of Segmentation: While the method aims to reduce ambiguity, the paper acknowledges that the segmentation of compound nouns can still be problematic due to inherent linguistic complexities.
  • Dictionary Size and Coverage: The effectiveness of the dictionary-building approach is dependent on the size and representativeness of the document set used.
  • Future Work: The authors suggest that further research could explore more sophisticated methods for identifying and evaluating candidate index terms, potentially incorporating more linguistic or semantic information. The efficiency and accuracy achieved by the proposed method make it a promising candidate for practical implementation in Korean information retrieval systems.

Summary updated

BibTeX
@inproceedings{park1996probabilistic,
  author = {Hyouk R. Park and Young S. Han and Kang H. Lee},
  title = {{A Probabilistic Approach to Compound Noun Indexing in Korean Texts}},
  booktitle = {International Conference on Computational Linguistics},
  year = {1996},
}
Figure