Open Domain New Word Detection Using Condition Random Field Method
Lists
Only you see this
Abstract
PDF HTML阅读 XML下载 导出引用 引用提醒 基于条件随机场方法的开放领域新词发现 DOI: 10.3724/SP.J.1001.2013.04254 作者: 作者单位: 作者简介: 通讯作者: 中图分类号: 基金项目: 国家自然科学基金(60903107, 61073071); 国家高技术研究发展计划(863)(2011AA01A205) Open Domain New Word Detection Using Condition Random Field Method Author: Affiliation: Fund Project: 摘要 | 图/表 | 访问统计 | 参考文献 | 相似文献 | 引证文献 | 资源附件 | 文章评论 摘要:开放领域新词发现研究对于中文自然语言处理的性能提升有着重要的意义.利用条件随机场(condition random field,简称CRF)可对序列输入标注的特点,将新词发现问题转化为预测已分词词语边界是否为新词边界的问题.在对海量规模中文互联网语料进行分析挖掘的基础上,提出了一系列区分新词边界的统计特征,并采用CRF方法综合这些特征实现了开放领域新词发现的算法,同时比较了K-Means 聚类、等频率、基于信息增益这3 种离散化方法对新词发现结果的影响.通过在SogouT 大规模中文语料库上的新词发现实验,验证了所提出的方法有较好的效果. Abstract:Open domain new word detection is vital for Chinese natural language processing research. This paper proposes a novel detection algorithm based condition random field (CRF), which treats the new word detection problem as a classification problem. In this algorithm, the study tries to separate boundaries of new words from existing words with both the CRF method and a serial of statistical features extracted from large scale corpus. The effectiveness of three different discretization strategies are also compared including K-means, equal-frequency, and information gain. Experimental results on a large-scale Web corpus named SogouT show the effectiveness of the proposed algorithms. 参考文献 相似文献 引证文献
BibTeX
@article{chen2013open,
author = {Fei Chen and Yiqun Liu and Chao Wei and Liang Yu and Min Zhang and Shaoping Ma},
title = {{Open Domain New Word Detection Using Condition Random Field Method}},
journal = {Journal of Software},
year = {2013},
doi = {10.3724/sp.j.1001.2013.04254},
}Loading related content...