End-To-End Speech Recognition Using A High Rank LSTM-CTC Based Model

Yangyang Shi, Mei-Yuh Hwang, Xin Lei

TL;DR

This paper introduces a "high rank projection layer" for end-to-end speech recognition models, aiming to improve their expressive power. By replacing a single bottleneck projection matrix with a more complex structure involving multiple projection matrices and non-linear activations, the model achieves significant reductions in word error rates on standard benchmarks, outperforming previous CTC-based methods without external data augmentation.

What Problem Does This Paper Address?

Traditional hybrid speech recognition systems involve a multi-step process, including acoustic modeling and alignment, which can be inefficient. End-to-end (E2E) models have emerged to simplify this by directly mapping acoustic features to output tokens. Among E2E approaches, Connectionist Temporal Classification (CTC) based models are popular due to their training simplicity and decoding efficiency. However, standard LSTM-CTC models often suffer from a bottleneck in the projection layer that maps LSTM hidden states to the final output probabilities. This layer, typically a single matrix, limits the model's ability to capture the full expressiveness of the LSTM features, especially when the output label set is relatively small compared to the LSTM's hidden state dimension. This paper aims to alleviate this bottleneck and enhance the model's performance.

Methods

The core innovation is the high rank projection layer. In conventional LSTM-CTC models, a single projection matrix $M$ maps the LSTM hidden feature vector $h_i$ to a logit vector $l_i$, which is then passed through a softmax to get output probabilities. This paper proposes replacing $M$ with a structure that:

  1. Multiple Projections: Maps the input hidden feature vector $h_i$ to $n$ different vectors ($l_{i,j}$) using $n$ distinct projection matrices ($M_j$).
  2. Non-linear Activation: Applies a non-linear activation function (specifically, tanh) to these intermediate vectors. This is crucial for breaking potential linear correlations between the projected vectors and increasing the "rank" of the projection.
  3. Weighted Interpolation: Combines these $n$ intermediate vectors using a learned weight matrix $W$. The weights $W_{i,j}$ are derived from $h_i$ via another projection and a softmax function, creating a weighted average.
  4. Temperature Factor: A temperature factor $\lambda$ is applied to the interpolated vector before the final softmax. This factor controls the smoothness of the output probability distribution, with higher values (e.g., 10-20) used to sharpen the distribution and make it more discriminative.

The entire high rank projection layer, including the projection matrices ($M_j$) and the weight matrix ($W$), is trained jointly with the rest of the network.

Datasets

The proposed model was evaluated on two standard speech recognition datasets:

  • Wall Street Journal (WSJ): Approximately 81 hours of transcribed speech audio. The data was split into training (95%), validation (5%), development (dev93), and evaluation (eval92) sets.
  • LibriSpeech: A large corpus of approximately 1000 hours of read speech based on public domain audiobooks. Similar to WSJ, a portion was used for training and validation, with "clean" and "other" subsets for development and evaluation.

The paper notes that for these experiments, no external data or data augmentation was applied, which is a significant factor for fair comparison.

Key Insights and Ideas

  • Bottleneck Mitigation: The primary insight is that the single projection matrix in standard LSTM-CTC models is a significant bottleneck. By introducing a more complex, higher-rank projection mechanism, the model can better leverage the rich information encoded in LSTM hidden states.
  • Importance of Non-linearity: The inclusion of a non-linear activation function (tanh) within the high rank projection layer is critical. Without it, the proposed method degenerates into a simpler mixture-of-matrices approach that does not yield improvements. This highlights that simply having multiple linear projections isn't enough; non-linearity is key to unlocking higher expressiveness.
  • Sharpening Output Distributions: The temperature factor $\lambda$ provides a mechanism to control the output distribution's sharpness. Using it to sharpen the distribution (higher $\lambda$) helps in making the predictions more discriminative, which is beneficial for tasks like speech recognition.
  • E2E Simplicity: The paper reinforces the advantage of E2E models by demonstrating competitive results without relying on external data, complex alignment procedures, or language model integration during training (though language models are used for decoding).

Key Results

The proposed high rank LSTM-CTC model achieved notable improvements on both datasets:

  • WSJ Corpus:
    • dev93: 6% relative Word Error Rate (WER) reduction compared to the baseline LSTM-CTC model.
  • eval92: 4% relative WER reduction compared to the baseline LSTM-CTC model.
  • LibriSpeech Corpus:
    • test-clean: 6% relative WER reduction.
  • test-other: 4% relative WER reduction.
  • Similar improvements were observed on the dev sets.

The paper also showed that removing the non-linear activation and temperature factor from the high rank projection layer (resulting in the "our-MOM-LSTM-CTC" model) did not improve performance, underscoring the importance of these components. The proposed model outperformed other published CTC-based end-to-end models under the condition of no external data or augmentation.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, a high rank projection layer with $n$ matrices will inherently have more parameters and potentially higher computational cost during inference compared to a single projection matrix. The paper does not quantify this trade-off.
  • Parameter Tuning: The choice of $n$ (the number of projection matrices) and the temperature factor $\lambda$ are hyperparameters that need to be tuned. The paper sets $n$ to be equal to the output label size to achieve the "highest rank," but exploring other values of $n$ could be a future direction.
  • Decoding: The paper mentions using WFST-based decoding with language models, following EESEN's approach. Future work could explore more advanced decoding strategies or end-to-end integration of language models.
  • Further Exploration of Rank: The paper focuses on increasing the "rank" of the projection. Further theoretical and empirical investigations into how different rank configurations affect performance could be beneficial.

Summary updated

Abstract

Long Short Term Memory Connectionist Temporal Classification (LSTM-CTC) based end-to-end models are widely used in speech recognition due to its simplicity in training and efficiency in decoding. In conventional LSTM-CTC based models, a bottleneck projection matrix maps the hidden feature vectors obtained from LSTM to softmax output layer. In this paper, we propose to use a high rank projection layer to replace the projection matrix. The output from the high rank projection layer is a weighted combination of vectors that are projected from the hidden feature vectors via different projection matrices and non-linear activation function. The high rank projection layer is able to improve the expressiveness of LSTM-CTC models. The experimental results show that on Wall Street Journal (WSJ) corpus and LibriSpeech data set, the proposed method achieves 4%-6% relative word error rate (WER) reduction over the baseline CTC system. They outperform other published CTC based end-to-end (E2E) models under the condition that no external data or data augmentation is applied. Code has been made available at https://github.com/mobvoi/lstm_ctc.
BibTeX
@inproceedings{shi2019end,
  author = {Yangyang Shi and Mei-Yuh Hwang and Xin Lei},
  title = {{End-To-End Speech Recognition Using A High Rank LSTM-CTC Based Model}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  doi = {10.1109/icassp.2019.8683297},
}
Figure