Abstract
Extracting nouns from Korean text is a complex task due to the language's agglutinative structure, where nouns are typically combined with various functional morphemes within a single spacing unit (Eojeol). Traditional rule-based morphological analyzers suffer from high maintenance costs and the "unknown word" problem. This paper proposes a statistical, syllable-based word recognition model that treats noun extraction as a sequence-tagging problem. By utilizing a Hidden Markov Model (HMM) and redefining the "word" unit to simplify analysis, the model achieves an F-measure of 91.11%, rivaling complex rule-based systems without requiring extensive linguistic dictionaries or manual rules.
Introduction: The Challenge of Korean Noun Extraction
In the Korean language, nouns rarely appear in isolation. As an agglutinative language, Korean constructs meaning by attaching postpositions, suffixes, and particles to a root noun. These combinations form an Eojeol, the basic unit separated by spaces in written text.
Traditional extraction methods rely on deep morphological analysis, which requires massive dictionaries and hand-crafted linguistic rules to "de-glue" these components. This approach faces two major hurdles: the "Unknown Word" problem, where the analyzer fails if a noun is missing from its dictionary, and high maintenance costs due to the expert labor required to update linguistic rules. This paper introduces a data-driven alternative that identifies noun boundaries based on the statistical patterns of syllables.
Methodology
The proposed Syllable-Based Word Recognition Model leverages a Hidden Markov Model (HMM) framework to bypass traditional morphological complexity.
Redefining the "Word"
To simplify the extraction task, the authors redefine a "word" as either a single uninflected morpheme (such as a noun) or a sequence of inflected morphemes. This abstraction avoids the intricate internal logic of traditional morphological analysis while remaining effective for noun identification.
Syllable Tagging Schemes
The model assigns tags to each syllable to denote its position and part-of-speech (POS). Four representation schemes were evaluated:
- BI (Begin, Inside)
- BIS (Begin, Inside, Single)
- IE (Inside, End)
- IES (Inside, End, Single)
The "Single" (S) tag specifically identifies one-syllable words, which are common in Korean particles and certain nouns.
Statistical Inference
The model employs a modified Markov assumption that incorporates word-spacing information. It calculates the most probable sequence of tags for a given string of syllables using the Viterbi algorithm. Training is automated by converting existing morpheme-level POS-tagged corpora into the new syllable-tag format, eliminating the need for manual rule-writing.
Datasets and Evaluation
The study utilized two primary resources:
- 21st Century Sejong Project Corpus: Millions of Eojeols from 1999–2001 used for training.
- ETRI POS Tagged Corpus: 288,269 Eojeols used as the test set to evaluate accuracy.
Results and Discussion
The performance was measured using the F-measure, balancing precision and recall.
Key Insights
- Syllables as Information Units: The model successfully exploits the fact that Korean syllables carry significant POS information. Certain syllables are statistically predisposed to start nouns or act as postpositions.
- The Importance of "Single" Tags: As shown in the results, the BIS and IES schemes outperformed the simpler BI/IE schemes. This indicates that accurately identifying short, one-syllable units is critical for high-performance Korean NLP.
- Contextual Tagging: Using a tagset that includes surrounding POS information (Tagset 2) yielded better results than a simplified "noun vs. non-noun" approach, proving that syntactic context from neighboring particles is essential for noun identification.
Performance Comparison
The BIS-2 configuration achieved a top F-measure of 91.11% (considering word frequency). This is comparable to HanTag (90.15%), a well-known POS tagger that relies on a complex morphological analyzer. Crucially, this model achieves these results without manually constructed linguistic knowledge.
Conclusion
This syllable-based approach offers a robust and efficient alternative to traditional morphological analysis for Korean noun extraction. By shifting from rigid rules to statistical syllable patterns, the model overcomes the unknown word problem and reduces maintenance overhead. Future work will focus on handling colloquial contractions and extending the model's context to further improve performance in diverse linguistic domains.
Summary updated