Chinese Unknown Word Identification Using Class-Based LM

Guohong Fu, Kang-Kwong Luke

Abstract

This paper presents a modified class-based LM approach to Chinese unknown word identification. In this work, Chinese unknown word identification is viewed as a classification problem and the part-of-speech of each unknown word is defined as its class. Furthermore, three types of features, including contextual class feature, word juncture model and word formation patterns, are combined in a framework of class-based LM to perform correct unknown word identification on a sequence of known words. In addition to unknown word identification, the class-based LM approach also provides a solution for unknown word tagging. The results of our experiments show that most unknown words in Chinese texts can be resolved effectively by the proposed approach.
BibTeX
@inproceedings{fu2004chinese,
  author = {Guohong Fu and Kang-Kwong Luke},
  title = {{Chinese Unknown Word Identification Using Class-Based LM}},
  booktitle = {IJCNLP},
  year = {2004},
  doi = {10.1007/978-3-540-30211-7\_74},
}

Loading related content...

Figure