Abstract
In this paper, we present a novel Deep Triphone Embedding (DTE) representation derived from Deep Neural Network (DNN) to encapsulate the discriminative information present in the adjoining speech frames. DTEs are generated using a four hidden layer DNN with 3000 nodes in each hidden layer at the first-stage. This DNN is trained with the tied-triphone classification accuracy as an optimization criterion. Thereafter, we retain the activation vectors (3000) of the last hidden layer, for each speech MFCC frame, and perform dimension reduction to further obtain a 300 dimensional representation, which we termed as DTE. DTEs along with MFCC features are fed into a second-stage four hidden layer DNN, which is subsequently trained for the task of tied-triphone classification. Both DNNs are trained using tri-phone labels generated from a tied-state triphone HMM-GMM system, by performing a forced-alignment between the transcriptions and MFCC feature frames. We conduct the experiments on publicly available TED-LIUM speech corpus. The results show that the proposed DTE method provides an improvement of absolute 2.11% in phoneme recognition, when compared with a competitive hybrid tied-state triphone HMM-DNN system.
Introduction
Speech recognition is fundamentally a sequence modeling problem where the identity of a phoneme is heavily influenced by its temporal context. While Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks are designed to capture these dependencies, they often present challenges such as vanishing gradients, high memory overhead, and sequential computation bottlenecks.
Standard feed-forward Deep Neural Networks (DNNs) are more efficient to train but lack an inherent mechanism for temporal memory. A common approach is to use a wide window of raw Mel-frequency cepstral coefficients (MFCCs) as input. However, this provides the network with raw data rather than a learned, high-level representation of context. This paper introduces Deep Triphone Embeddings (DTE), a method to provide feed-forward networks with contextual benefits similar to LSTMs while maintaining the simplicity of DNN training.
Methodology
The proposed DTE framework utilizes a two-stage learning process to extract and leverage contextual information.
Stage 1: Learning the Embedding
A first-stage DNN with four hidden layers (3000 nodes each) is trained to classify tied-triphones. Tied-triphones are speech units that represent a phoneme within its specific phonetic context. After training, the output layer is discarded, and the 3000-dimensional activation vector from the final hidden layer is extracted for each MFCC frame.
Dimensionality Reduction
To create a more compact and manageable representation, the 3000-dimensional vectors undergo dimensionality reduction using either Principal Component Analysis (PCA) or Linear Discriminant Analysis (LDA). This results in a 300-dimensional Deep Triphone Embedding (DTE).
Stage 2: Final Classification
A second four-layer DNN is then trained for the final classification task. The input to this network consists of the current frame's MFCCs concatenated with the DTEs of the preceding and succeeding frames. This architecture allows the second network to process a highly discriminative summary of the surrounding context.
Experimental Setup
The method was evaluated on the TED-LIUM speech corpus, which consists of transcribed TED talks. The training and development set included approximately 42 hours of speech (300 talks), while the test set comprised 11 talks (~2 hours). The lexicon included 152,000 words mapped to 40 phonemes and 1,373 tied-state triphones.
Results and Discussion
The DTE-based approach demonstrated significant improvements over traditional hybrid HMM-DNN systems.
Feature Disentanglement
A key advantage of DTE is its ability to disentangle phonetic features. Using t-SNE visualization, the authors compared the distribution of raw MFCCs with DTEs. While MFCCs often show overlapping clusters for different triphones, DTEs provide a much clearer separation in the feature space, indicating that the embeddings capture more discriminative contextual information.
Performance Metrics
The proposed method achieved a phoneme accuracy of 70.22% using DTE-PCA, representing a 6.72% absolute improvement over the standard HMM-DNN baseline (63.50%) and a 2.11% absolute improvement over a competitive wide-window, 8-layer DNN (68.11%).
| Training Method | Phoneme Accuracy (%) |
|---|---|
| HMM + GMM (Baseline) | 55.72 |
| HMM + DNN (Standard) | 63.50 |
| HMM + DNN (Wide Window + 8 Layers) | 66.11 |
| HMM + DTE-PCA + DNN (Proposed) | 70.22 |
The results suggest that providing a learned contextual representation is more effective than simply increasing the depth of a standard DNN.
Conclusion
Deep Triphone Embeddings offer a robust way to incorporate temporal context into feed-forward neural networks for speech recognition. By leveraging a two-stage training process and dimensionality reduction, DTEs provide a "pseudo-memory" that significantly enhances phoneme recognition accuracy. Future work will involve scaling this approach to larger datasets and integrating DTEs into Large Vocabulary Continuous Speech Recognition (LVCSR) systems to evaluate their impact on word error rates.
Summary updated