A novel unsupervised method for new word extraction

Lili Mei, Heyan Huang, Xiaochi Wei, Xianling MaoScience China Information Sciences 2016 · DOI · OpenReview

Abstract

New words could benefit many NLP tasks such as sentence chunking and sentiment analysis. However, automatic new word extraction is a challenging task because new words usually have no fixed language pattern, and even appear with the new meanings of existing words. To tackle these problems, this paper proposes a novel method to extract new words. It not only considers domain specificity, but also combines with multiple statistical language knowledge. First, we perform a filtering algorithm to obtain a candidate list of new words. Then, we employ the statistical language knowledge to extract the top ranked new words. Experimental results show that our proposed method is able to extract a large number of new words both in Chinese and English corpus, and notably outperforms the state-of-the-art methods. Moreover, we also demonstrate our method increases the accuracy of Chinese word segmentation by 10% on corpus containing new words.
BibTeX
@article{mei2016novel,
  author = {Lili Mei and Heyan Huang and Xiaochi Wei and Xianling Mao},
  title = {{A novel unsupervised method for new word extraction}},
  journal = {Science China Information Sciences},
  year = {2016},
  doi = {10.1007/s11432-015-0906-9},
}

Loading related content...

Figure