TL;DR
This paper presents an attention-based end-to-end speech recognition system specifically for Mandarin voice search. By employing character embeddings, regularization techniques, and attention smoothing, the authors achieve state-of-the-art results on a challenging Mandarin voice search dataset, significantly outperforming previous attempts.
What Problem Does This Paper Address?
Automatic Speech Recognition (ASR) is a crucial component for voice search, directly impacting user experience. While end-to-end ASR models have shown promise, applying them to Mandarin presents unique challenges. Mandarin's logographic nature, large vocabulary, and the conditional dependencies inherent in attention mechanisms make it difficult for these models to converge effectively. Previous attempts using joint character-pinyin models yielded high error rates. This paper aims to improve upon the Listen, Attend and Spell (LAS) architecture for Mandarin voice search by directly transcribing speech to characters, addressing the vocabulary and convergence issues.
Methods
The core of the proposed system is the Listen, Attend and Spell (LAS) architecture, an attention-based encoder-decoder model.
- Encoder (Listen): A bidirectional Long Short-Term Memory (BLSTM) recurrent neural network (RNN) processes the input acoustic speech features (Mel-scale filterbank coefficients) to generate a high-level representation sequence.
- Attention Mechanism (Attend): This module dynamically selects relevant encoder features to predict the next output character. The paper compares two types of attention:
- Content-based attention: A standard approach where the context vector is a weighted sum of encoder hidden states.
- Location-based attention: An extension that incorporates alignment information from the previous step to improve robustness.
- Decoder (Spell): A unidirectional LSTM RNN predicts the output characters based on the attended context vector and the previously predicted character.
- Character Embedding: To handle Mandarin's large vocabulary, each character is represented by a one-hot encoding and then mapped to a dense embedding vector using a fully-connected neural network layer. This layer's weights are trained jointly with the rest of the LAS model.
- Regularization Techniques: To improve model convergence and generalization, the authors employ:
- L2 regularization: Penalizes large weights.
- Gaussian weight noise: Adds noise to weights during training.
- Frame skipping: A technique used during training to speed up BLSTM processing by skipping frames.
- Attention Smoothing: To address the issue of attention focusing too narrowly on a few frames, the authors replace the softmax function in the attention weight calculation with a logistic sigmoid function. This aims to maintain diversity and smooth the attention focus.
- Language Model Integration: For decoding, a character-level language model (built from a word-level LM and lexicon) is integrated to further refine predictions.
Datasets
- Training Data: A 3000-hour dataset of Mandarin voice search utterances collected from the MiTV remote controller. This dataset comprises approximately 4 million voice search queries.
- Test and Validation Data: Held-out validation and test sets were also collected from the MiTV voice search, each consisting of 3,000 utterances.
- Vocabulary: The decoder model uses 6,925 labels, including 6,922 common Chinese characters, an unknown token, and sentence start/end tokens.
Key Insights and Ideas
- Character Embeddings are Crucial for Mandarin: Directly using character embeddings, rather than more complex joint models, proved effective for handling Mandarin's large vocabulary and improving model convergence.
- Attention Smoothing Enhances Robustness: Replacing softmax with a sigmoid in the attention mechanism helps to maintain attention diversity and smooth the focus, leading to better performance, especially on longer contexts.
- Regularization is Key for Generalization: Techniques like L2 regularization, Gaussian weight noise, and frame skipping are essential for training robust end-to-end models that generalize well.
- Language Model Integration Provides Further Gains: While the attention-based model performs well on its own, integrating a trigram language model during decoding yields significant improvements in accuracy.
Key Results
The paper reports impressive results on the MiTV voice search dataset:
- Without Language Model:
- Content-based attention model: CER of 4.05%, SER of 9.1%.
- Location-based attention model: CER of 3.82%, SER of 8.17%.
- Attention smoothing model: Achieved a CER of 3.58% and an SER of 7.43%.
- With Trigram Language Model:
- Attention smoothing + trigram LM: Achieved the lowest CER of 2.81% and SER of 5.77%.
The authors highlight that attention smoothing provided an 11.6% relative gain in CER over the content-based attention model.
Limitations and Future Work
- Dataset Specificity: The authors acknowledge that the MiTV voice search dataset is from a "limited with closed domains" context.
- Future Work: The authors plan to investigate their approach on more general ASR tasks using public datasets. They also suggest further exploration of their attention-based model for broader ASR applications.
Summary updated