A novel unsupervised method for new word extraction
Lists
Only you see this
Abstract
New words could benefit many NLP tasks such as sentence chunking and sentiment analysis. However, automatic new word extraction is a challenging task because new words usually have no fixed language pattern, and even appear with the new meanings of existing words. To tackle these problems, this paper proposes a novel method to extract new words. It not only considers domain specificity, but also combines with multiple statistical language knowledge. First, we perform a filtering algorithm to obtain a candidate list of new words. Then, we employ the statistical language knowledge to extract the top ranked new words. Experimental results show that our proposed method is able to extract a large number of new words both in Chinese and English corpus, and notably outperforms the state-of-the-art methods. Moreover, we also demonstrate our method increases the accuracy of Chinese word segmentation by 10% on corpus containing new words.
BibTeX
@article{mei2016novel,
author = {Lili Mei and Heyan Huang and Xiaochi Wei and Xianling Mao},
title = {{A novel unsupervised method for new word extraction}},
journal = {Science China Information Sciences},
year = {2016},
doi = {10.1007/s11432-015-0906-9},
}Loading related content...