TL;DR
This paper introduces a novel two-step approach for segmenting compound nouns in Korean, a language where words are often written without spaces, making constituent identification challenging. By combining a manually curated dictionary of frequent compound nouns with a statistical segmentation algorithm based on lexical frequencies, the system achieves high accuracy, crucial for downstream NLP tasks like machine translation and information retrieval.
What Problem Does This Paper Address?
Korean is an agglutinative language where words are frequently written without spaces, forming "eojeols" that can contain multiple content and functional words. This lack of explicit word boundaries poses a significant challenge for natural language processing (NLP) tasks, particularly for identifying compound nouns. Unlike English, Korean lacks clear capitalization cues to distinguish proper nouns, further complicating segmentation. The problem is that it's impossible to pre-register every possible compound noun due to their sheer number and the open-ended nature of word formation. Without accurate segmentation, understanding the meaning of sentences and phrases, retrieving relevant documents, or performing accurate machine translation becomes difficult. For instance, to retrieve documents about "Seol'ag Mountain," a system needs to segment "seol'agsan-gugrib-gongwon" (Seol'ag Mountain National Park) correctly.
Methods
The paper proposes a two-step method for compound noun segmentation:
- Hand-Built Segmentation Dictionary (HBSD): This involves extracting a large corpus (30 million words) and using a simple noun dictionary to identify potential compound nouns. These candidates are then manually analyzed and elaborated to create a dictionary of frequently used compound nouns that require special handling. This dictionary contains approximately 100,000 entries.
- Statistical Segmentation Algorithm: For compound nouns not found in the HBSD, a statistical algorithm is employed. This algorithm relies on the frequencies of individual simple nouns extracted from a corpus. The core of this algorithm is a modification of the CYK (Cocke-Younger-Kasami) parsing method, adapted to a bottom-up approach. It utilizes a "min-max composition" strategy to determine the most likely segmentation. This involves calculating the minimum probability of combining two segments and then taking the maximum of these minimums. The algorithm iteratively builds up segmentation possibilities for sequences of syllables, ultimately determining the most probable segmentation for the entire compound noun.
Datasets
- 30 million word corpus: Used for initial extraction of potential compound nouns and for building the simple noun dictionary.
- ETRI POS tagged corpus: A corpus of approximately 270,000 eojeols used for extracting compound nouns for the gold standard test set.
- KAIST Tree Bank Project data: Supplemented the HBSD.
Key Insights and Ideas
- The importance of lexical data: The paper strongly emphasizes that for compound noun segmentation, lexical information (frequencies of individual nouns) is far more critical than part-of-speech transition probabilities, which are less useful in a language like Korean.
- Min-Max Composition: This novel approach for combining segmentation possibilities is key. It prioritizes segmentations where both resulting parts are likely words, rather than just focusing on the overall probability of the combined segment. This helps avoid oversegmentation.
- Bottom-up CYK parsing for segmentation: Adapting a parsing technique for segmentation is an elegant way to systematically explore all possible segmentation combinations and build up to the final result.
- The value of "distinct nouns": Identifying frequently occurring nouns ("distinct nouns") is crucial for building a robust segmentation system, as these are often the building blocks of compound nouns.
- The trade-off between dictionary size and performance: The paper notes that including too many rarely used nouns in a dictionary can lead to oversegmentation, highlighting the need for careful selection.
Key Results
- Overall Segmentation Accuracy (SA): The proposed system achieved an accuracy of 97.29% on the gold standard test set.
- Precision and Recall: The system achieved high precision (98.04%) and recall (97.80%) for constituent segmentation.
- Baseline vs. Whole System: The baseline system (using only distinct nouns and the algorithm) achieved 94.30% SA, while the whole system (including the HBSD) improved this to 97.29%, demonstrating the significant impact of the built-in dictionary.
Limitations and Future Work
- Unsegmented Syllables: The paper acknowledges that some syllables may remain unsegmented. This can occur due to:
- Unregistered words within a compound noun.
- Combinations of proper nouns with positional or geometric information.
- Proper nouns themselves (especially transliterated foreign words).
- Future Work: The paper suggests that the system is promising and can be helpful for applications like machine translation and information retrieval. The current work focuses on improving the segmentation accuracy, and further integration into these systems would be a natural next step. The default analysis and tuning section hints at strategies for handling unsegmented strings, which could be further developed.
Summary updated