Large Vocabulary Speech Recognition Using Deep Tensor Neural Networks

Dong Yu, Li Deng, Frank Seide

Introduction

In the landscape of 2012 speech recognition, Context-Dependent Deep Neural Network Hidden Markov Models (CD-DNN-HMMs) were the state-of-the-art. However, standard Deep Neural Networks (DNNs) primarily rely on simple linear projections followed by element-wise nonlinearities. This paper argues that such architectures struggle to efficiently model the complex, multi-way interactions inherent in speech, such as the interplay between speaker identity, background noise, and phonetic content. To address this, the authors introduce Deep Tensor Neural Networks (DTNNs), which utilize specialized layers to explicitly capture these latent factor interactions.

Architecture: Modeling Multi-Way Interactions

The core innovation of the DTNN is the replacement of standard hidden layers with a combination of Double-Projection (DP) and Tensor layers.

  1. Double-Projection (DP) Layer: Instead of a single projection, the input is mapped into two separate nonlinear subspaces ($h_1$ and $h_2$).
  2. Tensor Layer: This layer models the interactions between these two subspaces using a three-way tensor. Every output neuron represents a weighted combination of all possible products between elements of the two subspaces.

To overcome the computational difficulty of training tensors, the authors employ a "vectorization trick." By defining the input to the subsequent layer as the cross product of the two subspaces ($v = h_1 \otimes h_2$), the tensor operation is mathematically rearranged into a standard matrix multiplication. This allows the model to be trained using standard backpropagation within existing deep learning frameworks.

Figure 3: Comparison of (a) a standard DNN layer, (b) a double-projection layer, and (c) the vectorized tensor layer implementation.

Experimental Results

The models were evaluated on a 30-hour subset of the Switchboard I training set, with testing performed on the Switchboard portion of the NIST 2000 Hub5 evaluation set. A trigram language model trained on the 2000h Fisher-corpus transcripts was used for decoding.

The researchers compared a baseline 5-layer DNN against various DTNN configurations. The results demonstrated that the DTNN architecture provides a significant performance boost over standard DNNs.

Figure 4: Word Error Rate (WER) comparison between the baseline CD-GMM-HMM, the baseline DNN (2kx5), and various DTNN configurations.

As shown in the results, the baseline DNN achieved a 27.4% WER. The best-performing DTNN configuration (2kx4-(96:96)x1) reached a 26.2% WER, representing a ~5% relative reduction in error. Notably, these gains were achieved without the generative pre-training (using Restricted Boltzmann Machines) typically required for DNNs at the time, highlighting the inherent robustness of the DTNN architecture.

Key Insights and Discussion

  • Strategic Placement: The authors found that replacing the top hidden layer (closest to the output) with a DP/Tensor layer was more effective than replacing bottom layers. They hypothesize that tensor layers operate better on the "binary-like" activations of higher-level sigmoid layers than on raw, real-valued acoustic features.
  • Efficiency: DTNNs can achieve higher accuracy than standard DNNs with a similar number of parameters, suggesting that the tensor structure is a more efficient allocation of modeling power for speech tasks.
  • Limitations: Training DTNNs remains computationally expensive. The lack of efficient cross-machine parallel algorithms limited the study to a 30-hour subset of the Switchboard data. Future work will focus on scaling these models to larger datasets and exploring the impact of generative pre-training specifically designed for tensor structures.

Summary updated

BibTeX
@inproceedings{yu2012large,
  author = {Dong Yu and Li Deng and Frank Seide},
  title = {{Large Vocabulary Speech Recognition Using Deep Tensor Neural Networks}},
  booktitle = {Interspeech},
  year = {2012},
  doi = {10.21437/interspeech.2012-2},
}
Figure