Unsupervized Word Segmentation: the case for Mandarin Chinese
Lists
Only you see this
Abstract
In this paper, we present an unsupervized segmentation system tested on Mandarin Chinese. Following Harris's Hypothesis in Kempe (1999) and Tanaka-Ishii's (2005) reformulation, we base our work on the Variation of Branching Entropy. We improve on (Jin and Tanaka-Ishii, 2006) by adding normalization and viterbidecoding. This enable us to remove most of the thresholds and parameters from their model and to reach near state-of-the-art results (Wang et al., 2011) with a simpler system. We provide evaluation on different corpora available from the Segmentation bake-off II (Emerson, 2005) and define a more precise topline for the task using cross-trained supervized system available off-the-shelf (Zhang and Clark, 2010; Zhao and Kit, 2008; Huang and Zhao, 2007)
BibTeX
@inproceedings{magistry2012unsupervized,
author = {Pierre Magistry and Benoît Sagot},
title = {{Unsupervized Word Segmentation: the case for Mandarin Chinese}},
booktitle = {Annual Meeting of the Association for Computational Linguistics},
year = {2012},
url = {https://openreview.net/forum?id=H1E1ajedbr},
}Loading related content...