Entropy as an Indicator of Context Boundaries: An Experiment Using a Web Search Engine

Kumiko Tanaka-Ishii

Abstract

Previous works have suggested that the uncertainty of tokens coming after a sequence helps determine whether a given position is at a context boundary. This feature of language has been applied to unsupervised text segmentation and term extraction. In this paper, we fundamentally verify this feature. An experiment was performed using a web search engine, in order to clarify the extent to which this assumption holds. The verification was applied to Chinese and Japanese.
BibTeX
@inproceedings{tanakaishii2005entropy,
  author = {Kumiko Tanaka-Ishii},
  title = {{Entropy as an Indicator of Context Boundaries: An Experiment Using a Web Search Engine}},
  booktitle = {IJCNLP},
  year = {2005},
  doi = {10.1007/11562214\_9},
}

Loading related content...

Figure