Handling OOVWords in Mandarin Spoken Term Detection with an Hierarchical n‐Gram Language Model

Xuyang Wang, Pengyuan Zhang, Xingyu Na, Jielin Pan, Yonghong YanChinese 2017 · DOI

Abstract

In this paper, an hierarchical n-gram Language model (LM) combining words and characters is explored to improve the detection of Out-of-vocabulary (OOV) words in Mandarin Spoken term detection (STD). The hierarchical LM is based on a word-level LM, with a character-level LM estimating probabilities of OOV words in a class-based way. The region containing OOV words in the sentence to be decoded is detected with the help of the word-level LM and the probabilities of OOV words are derived from the character-level LM. The implementation of the proposed approach is based on a dynamic decoder. The proposed approach is evaluated in terms of Actual term weighted value (ATWV) on two Mandarin data sets. Experiment results show that more than 10% relative improvement for OOV word detection is achieved on both sets. In addition, the detection of In-vocabulary (IV) words is barely influenced as well.
BibTeX
@inproceedings{wang2017handling,
  author = {Xuyang Wang and Pengyuan Zhang and Xingyu Na and Jielin Pan and Yonghong Yan},
  title = {{Handling OOVWords in Mandarin Spoken Term Detection with an Hierarchical n‐Gram Language Model}},
  booktitle = {Chinese},
  year = {2017},
  doi = {10.1049/cje.2017.07.004},
}

Loading related content...

Figure