Recurrent neural network language model with structured word embeddings for speech recognition

Tianxing He, Xu Xiang, Yanmin Qian, Kai Yu

Abstract

Due to effective word context encoding and long-term context preserving, recurrent neural network language model (RNNLM) has attracted great interest by showing better performance over back-off n-gram models and feed-forward neural network language models (FNNLM). However, it still has the difficulty of modelling words of very low frequency in training data. To address this issue, a new framework of structured word embedding is introduced to RNNLM, where both input and target word embeddings are factorized into weighted sum of the corresponding sub-word embeddings. The framework is instantiated for Chinese, where characters can be naturally used as the sub-word units. Experiments on a Chinese twitter LVCSR task showed that the proposed approach effectively outperformed the standard RNNLM, yielding a relative PPL improvement of 8:8% and an absolute 0:59% CER improvement in N-Best re-scoring.
BibTeX
@inproceedings{he2015recurrent,
  author = {Tianxing He and Xu Xiang and Yanmin Qian and Kai Yu},
  title = {{Recurrent neural network language model with structured word embeddings for speech recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2015},
  doi = {10.1109/icassp.2015.7179002},
}

Loading related content...

Figure