A general artificial neural network extension for HTK

Chao Zhang, Philip C. Woodland

Introduction

For decades, the Hidden Markov Model Toolkit (HTK) served as the industry and academic standard for automatic speech recognition (ASR). However, the rapid rise of the "Deep Learning Revolution" after 2010 left a gap in the toolkit's native capabilities, as it lacked built-in support for Deep Neural Networks (DNNs). To implement hybrid models, researchers were often forced to migrate to newer toolkits like Kaldi or RWTH-ASR, or rely on external tools such as QuickNet.

This paper introduces HTK-ANN, a native artificial neural network extension designed to revitalize HTK. The extension provides a generic, compatible, and research-friendly environment that allows for the construction of flexible, high-performance hybrid and tandem ASR systems within the familiar HTK framework.

Technical Framework and Methods

The authors implemented several new modules—including HANNet for network structures and HNTrainSGD for training—while updating core HTK tools to support neural architectures.

Flexible Architectures

Unlike rigid, layer-stacked toolkits, HTK-ANN utilizes a "feature mixture" approach. Layers are not restricted to sequential connections; they can take inputs from any other layer or raw acoustic features using arbitrary "context shifts" (e.g., simultaneously processing frames at $t-1, t, t+2$). This flexibility allows researchers to define complex Directed Acyclic Graphs (DAGs) for their models.

Training and Optimization

The system supports standard frame-level Cross Entropy (CE) training as well as advanced sequence-level discriminative training criteria, such as Maximum Mutual Information (MMI) and Minimum Phone Error (MPE). To handle the computational demands of deep learning, the extension includes GPU support via CUDA and CPU optimization through Intel MKL. A robust 64-bit data cache with multiple shuffling modes (frame, utterance, and batch-utterance) ensures that data is randomized efficiently to keep the hardware fully utilized.

Hybrid and Tandem System Integration

A primary advantage of a native ANN extension is the seamless synergy with existing HTK tools. The toolkit can treat classic GMM-HMM transforms, such as Heteroscedastic Linear Discriminant Analysis (HLDA) or Constrained Maximum Likelihood Linear Regression (CMLLR), as integrated layers within the network.

Figure 2: Diagram of a Tandem ASR system where a Bottleneck DNN acts as a feature extractor integrated with classic GMM-HMM transforms like HLDA and CMLLR.

This integration is particularly powerful for "Tandem" systems, where a DNN serves as a sophisticated feature extractor for a traditional HMM. Additionally, the toolkit supports model stacking, allowing multiple small neural network modules to be layered to progressively refine acoustic features.

Experimental Results

The HTK-ANN extension was evaluated on two distinct tasks: the 15-hour Wall Street Journal (WSJ0) English task and the massive 300-hour DARPA BOLT Mandarin task, which involves spontaneous conversational telephone speech with multiple accents and mixed languages.

Performance on BOLT Mandarin

On the challenging BOLT task, the HTK-ANN hybrid system demonstrated state-of-the-art performance. The transition from standard CE training to sequence-level MPE training resulted in an 8.9% relative reduction in Word Error Rate (WER).

Figure 4: Comparison of Word Error Rates (WER) for different system configurations and training criteria on the BOLT Mandarin task.

As shown in the results, the hybrid system achieved a 31.6% WER. By employing joint decoding to combine the hybrid system with a tandem system (Tandem SAT), the authors further reduced the WER to 31.0%.

Conclusion

HTK-ANN represents a significant milestone in the toolkit's history, officially integrated into HTK version 3.5. By bridging the gap between classic HMM-based recognition and modern deep learning, it provides a stable and scalable platform for ASR research. Future work aims to expand the toolkit's capabilities to include native Backpropagation Through Time (BPTT) for Recurrent Neural Networks (RNNs) and the addition of convolutional layers.

Summary updated

BibTeX
@inproceedings{zhang2015general,
  author = {Chao Zhang and Philip C. Woodland},
  title = {{A general artificial neural network extension for HTK}},
  booktitle = {Interspeech},
  year = {2015},
  doi = {10.21437/interspeech.2015-710},
}
Figure