ExKaldi-RT: A Real-Time Automatic Speech Recognition Extension Toolkit of Kaldi

Yu Wang, Chee Siang Leow, Akio Kobayashi, Takehito Utsuro, Hiromitsu Nishizaki

TL;DR

This paper introduces ExKaldi-RT, a Python-based toolkit that extends Kaldi for real-time automatic speech recognition (ASR). It simplifies the development of online ASR systems by providing a flexible pipeline architecture, enabling researchers to integrate custom deep learning models and achieve competitive performance.

What Problem Does This Paper Address?

Building real-time automatic speech recognition (ASR) systems presents a significant challenge. While powerful offline ASR toolkits like Kaldi exist, they often lack the necessary components and flexibility for seamless integration into live, streaming applications. Existing extensions that bridge Kaldi with deep learning frameworks primarily focus on offline processing, leaving a gap for developers needing to create and evaluate online ASR systems under realistic conditions. This paper addresses the need for a user-friendly, Python-based toolkit that facilitates the construction and customization of real-time ASR pipelines, allowing for the integration of modern deep learning models.

Methods

ExKaldi-RT builds upon the established Kaldi ASR toolkit but offers a Python-centric approach for online ASR. Its core is a modular pipeline architecture, where "components" (e.g., audio recorders, feature extractors, decoders) are connected by "pipes." This design allows for:

  • Online Feature Extraction: It wraps Kaldi's tools for computing features like MFCCs and fBanks, and crucially, supports custom feature processing, including integrating deep learning models for tasks like speech separation.
  • Voice Activity Detection (VAD): Implements VAD to filter out silent segments, reducing unnecessary computation and improving efficiency.
  • Remote Transmission: Enables client-server architectures for distributed ASR processing.
  • Online Decoding: Integrates Kaldi's LatticeFasterDecoder but allows it to accept acoustic probabilities from custom deep learning models trained in frameworks like TensorFlow or PyTorch. This is a key differentiator, enabling the use of advanced DNNs (like TDNNs and LSTMs) for acoustic modeling.
  • Customization: A major emphasis is placed on allowing users to customize individual components, enabling them to implement novel algorithms for signal processing, feature extraction, and even N-best rescoring with RNN-based language models.

Datasets

  • LibriSpeech: The minimum LibriSpeech corpus was used for experiments, consisting of 5 hours of clean data for training and 2 hours for evaluation.
  • NOISEX-92: This dataset was used to synthesize noisy data for experiments involving speech separation.

Key Insights and Ideas

  • Pythonic Flexibility for Real-Time ASR: The toolkit's primary innovation is providing a Python interface to Kaldi's powerful ASR capabilities, specifically tailored for real-time applications. This lowers the barrier to entry for researchers and developers wanting to experiment with online ASR.
  • Seamless Deep Learning Integration: ExKaldi-RT bridges the gap between Kaldi's traditional acoustic modeling and modern deep learning frameworks. This allows for the use of highly accurate DNNs trained externally, directly within an online ASR pipeline.
  • Modularity is Key: The component-and-pipe architecture is a powerful design choice. It allows for easy swapping and customization of different parts of the ASR pipeline, fostering rapid prototyping and experimentation with novel techniques.
  • Speech Separation Enhances Robustness: The paper demonstrates that integrating a U-Net-based speech separation model significantly improves ASR performance in noisy environments, highlighting the toolkit's ability to incorporate advanced signal processing techniques.

Key Results

  • Competitive Performance: ExKaldi-RT achieved competitive ASR performance on the minimum LibriSpeech corpus, with a Word Error Rate (WER) of 19.97% in real-time, very close to the offline baseline's 18.49%.
  • Real-Time Factor (RTF): The toolkit achieved a real-time factor of 0.57, indicating that the system can process audio in real-time.
  • Feature Impact: Experiments showed that a mixture of MFCC, fBank, and LDA+MLLT features yielded the best WER (18.56%) at an acceptable RTF.
  • Speech Separation Gains: In noisy conditions, integrating a U-Net speech separation model reduced the WER by 9.49 points compared to using the acoustic model trained on clean data in a noisy environment.

Limitations and Future Work

  • Data Encoding: Currently, the toolkit provides a simple function for encoding data packets without compression, which could be improved for better transmission efficiency.
  • CPU Bottleneck: The paper notes that batch processing with multiple CPU threads can be a bottleneck for real-time performance, especially when multiple neural networks are involved.
  • Future Work: The authors plan to expand online decoders to support end-to-end (E2E) approaches, which represent a significant trend in modern ASR research.

Summary updated

Abstract

This paper describes the ExKaldi-RT online automatic speech recognition (ASR) toolkit that is implemented based on the Kaldi ASR toolkit and Python language. ExKaldi-RT provides tools for building online recognition pipelines. While similar tools are available built on Kaldi, a key feature of ExKaldi-RT that it works on Python, which has an easy-to-use interface that allows online ASR system developers to develop original research, such as by applying neural network-based signal processing and by decoding model trained with deep learning frameworks. We performed benchmark experiments on the minimum LibriSpeech corpus, and it showed that ExKaldi-RT could achieve competitive ASR performance in real-time recognition.
BibTeX
@inproceedings{wang2021exkaldi,
  author = {Yu Wang and Chee Siang Leow and Akio Kobayashi and Takehito Utsuro and Hiromitsu Nishizaki},
  title = {{ExKaldi-RT: A Real-Time Automatic Speech Recognition Extension Toolkit of Kaldi}},
  booktitle = {2021},
  year = {2021},
  doi = {10.1109/gcce53005.2021.9621992},
}
Figure