Chinese Unknown Word Identification Using Character-based Tagging and Chunking

Chooi Ling Goh, Masayuki Asahara, Yuji MatsumotoACL 2003

TL;DR

This paper tackles the challenge of identifying unknown words in Chinese text, a crucial step for many NLP tasks due to the absence of explicit word delimiters. They propose a novel two-stage approach: first, morphological analysis to segment words and assign part-of-speech (POS) tags, followed by a Support Vector Machine (SVM)-based chunker that leverages character-level features to detect unknown words.

What Problem Does This Paper Address?

Written Chinese, unlike many Western languages, does not use spaces to separate words. This makes word segmentation a fundamental preprocessing step for any Chinese Natural Language Processing (NLP) task. However, traditional word segmentation methods struggle when encountering "unknown words" – words not present in their dictionaries. This is a persistent problem because new words are constantly being created by combining existing characters. Simply expanding dictionaries is impractical. Therefore, accurately identifying these unknown words is essential for robust Chinese NLP.

Methods

The authors propose a unified approach consisting of three main steps:

  1. Morphological Analysis: They utilize a morphological analyzer (specifically mentioning ChaSen, adapted from Japanese NLP) to segment the Chinese text into words and assign Part-of-Speech (POS) tags. This step provides the initial word boundaries.
  2. Character-Based Features: To improve unknown word detection, especially when unknown words might be part of known words, they break down segmented words into individual characters. Each character is then assigned features, including its position within a word (e.g., beginning, inside, end) and potentially its type (e.g., family name, transliteration character). This character-level granularity is key.
  3. Chunking with Support Vector Machine (SVM): An SVM-based chunker (YamCha) is employed to identify unknown words. The chunker classifies characters into three categories: 'B' (beginning of a chunk), 'I' (inside a chunk), and 'O' (outside a chunk). A sequence of characters tagged as 'B' followed by 'I's is considered an unknown word (a chunk). The chunker uses a context window of two characters on each side of the current character for feature extraction. They explore both forward and backward parsing strategies.

Datasets

The experiments were conducted using a one-month news corpus from the People's Daily from 1998. This corpus contained approximately 300,000 words and 1,000,000 characters, with 39 POS tags. The corpus was randomly split into training and testing sets with a 4:1 ratio. For evaluation, person names and organization names were deliberately removed from the dictionary. Additionally, words that appeared only once in the corpus were also treated as unknown words to simulate general unknown word detection.

Key Insights and Ideas

  • Character-level granularity is crucial: By breaking down words into characters, the system can identify unknown words that might otherwise be missed if they are embedded within known words or if the segmentation is slightly off.
  • Leveraging morphological analysis as a first step: Using an existing morphological analyzer provides a solid foundation of segmented words and POS tags, which are then enhanced by the character-based chunking.
  • SVMs are effective for chunking: The SVM-based chunker, combined with character features, proves to be a powerful tool for identifying unknown word boundaries.
  • Backward parsing is beneficial for specific tasks: For organization name extraction, backward parsing yielded better results, likely because organization names often end with specific keywords (e.g., "company," "group").

Key Results

The proposed method achieved promising results across different categories of unknown words:

  • Person Name Extraction: Achieved an F-measure of 86.66% (with family name features and forward parsing) and 86.78% (with family name features and backward parsing). The system could also correctly extract names that overlapped with known words.
  • Organization Name Extraction: Backward parsing achieved an F-measure of 70.40%, outperforming forward parsing.
  • General Unknown Word Extraction: The system achieved an F-measure of 61.00% (with backward parsing), where unknown words included common nouns, verbs, and numbers that appeared only once in the corpus.
  • Comparison with Word-Based Chunking: Character-based chunking consistently outperformed word-based chunking across all categories, with F-measures for person names (84.69% vs 81.28%), organization names (70.40% vs 67.88%), and general unknown words (61.00% vs 56.96%).

Limitations and Future Work

  • Context Window Size: The authors note that for some organization names that are very long, the fixed two-character context window might not be sufficient for the system to make a correct judgment. This suggests that exploring larger or adaptive context windows could be a direction for future work.
  • Comparison with Other Works: While they compare their results to previous work, they acknowledge that direct comparison can be difficult due to differences in corpora and experimental setups.
  • Future Work: The paper implicitly suggests that further improvements could be made by refining the character features and potentially exploring more advanced machine learning models or hybrid approaches. The success in detecting person and organization names, and general unknown words, indicates the robustness of the character-based chunking approach.

Summary updated

BibTeX
@inproceedings{goh2003chinese,
  author = {Chooi Ling Goh and Masayuki Asahara and Yuji Matsumoto},
  title = {{Chinese Unknown Word Identification Using Character-based Tagging and Chunking}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2003},
}
Figure