Abstract
Traditional Deep Belief Networks (DBNs) for speech recognition are typically trained using frame-level criteria, which fail to capture the inherent sequential dependencies in speech. This paper proposes a full-sequence training approach that treats the top layer of a DBN as a linear-chain Conditional Random Field (CRF). This hybrid architecture allows for the joint optimization of deep non-linear feature weights and temporal transition parameters. Experiments on the TIMIT corpus demonstrate that sequence-level fine-tuning significantly outperforms frame-based baselines, achieving state-of-the-art results for phonetic recognition.
Introduction
In the early 2010s, DBNs emerged as a powerful tool for speech recognition, surpassing traditional Gaussian Mixture Models. However, these models were primarily trained on isolated frames or small windows of audio. Because speech is a temporal signal where phonetic states follow specific grammatical and acoustic rules, frame-level training is suboptimal. This research addresses the gap by optimizing the model over entire utterances, ensuring the predicted sequence as a whole is maximized.
Methodology
The proposed model combines a multi-layer DBN with a linear-chain CRF. The DBN serves as a deep feature extractor, while the CRF layer models the transitions between phonetic states.
The training process involves unsupervised generative pre-training of RBM layers followed by sequence-level fine-tuning. This fine-tuning maximizes the log conditional likelihood of the full phonetic sequence, jointly updating DBN weights and state-to-state transition parameters. Additionally, HMM-style transition constraints are enforced to maintain valid phonetic sequences.
Experimental Setup
The TIMIT corpus was used for evaluation, featuring 462 speakers for training and a core test set of 24 speakers. Audio was preprocessed into 12th-order MFCCs with energy and derivatives, normalized to zero mean and unit variance.
Results and Discussion
The study highlights the importance of joint optimization and proper initialization. For shallow models (3 layers), sequence training provided a modest boost. However, for deeper models (6 layers), the improvement was more pronounced but required careful hyperparameter tuning.
As shown in the table above, mini-batch size is a critical factor for the stability and performance of sequence-based training in deep networks. Larger mini-batches (e.g., 800) helped the 6-layer model achieve a core test accuracy of 77.81%, compared to the 77.15% frame-based baseline. Further refinement by re-initializing pre-training with sequence-learned weights pushed the accuracy to 78.25%.
Conclusion
Full-sequence training of DBNs effectively bridges the gap between deep feature learning and temporal sequence modeling. While computationally more demanding and sensitive to optimization parameters, the approach yields superior recognition performance. Future work will explore more complex objective functions and speaker adaptation techniques.
Summary updated