End-to-End Speech Endpoint Detection Utilizing Acoustic and Language Modeling Knowledge for Online Low-Latency Speech Recognition

Inyoung Hwang, Joon-Hyuk ChangIEEE Access 2020 · DOI

Introduction

Speech endpoint detection (EPD) is a critical component of voice user interfaces, responsible for determining exactly when a user has finished speaking. Accurate EPD is a balancing act: premature detection (early endpointing) cuts off users mid-sentence, while delayed detection (late endpointing) makes the system feel sluggish and unresponsive.

State-of-the-art EPD systems often rely on Decoder State Features (DSFs) from an Automatic Speech Recognition (ASR) engine. While effective, these features require running a full ASR decoding process in real-time, creating a significant computational burden for resource-constrained devices like smartwatches or embedded AI speakers. This research proposes a novel end-to-end EPD framework that captures the benefits of language modeling knowledge without the heavy "tax" of full decoding at runtime.

Proposed Framework

The core of the proposed system is an ensemble architecture using Recurrent Neural Networks (RNNs) that fuses three distinct types of knowledge:

  1. Acoustic Feature Embedding (AFE): A standard approach that analyzes raw spectral features to distinguish speech from silence.
  2. Phonetic Embedding (PE): This incorporates acoustic modeling knowledge by training a network to recognize "senones" (phonetic units), providing the system with phonetic context.
  3. LM-based EOU Predictor: The primary innovation of this work. Instead of a full decoder, a specific Language Model (LM)-based End-of-Utterance (EOU) predictor is trained. During training, it learns from N-gram probabilities derived from a 1-best decoding hypothesis. At inference time, it provides linguistic context independently of the full ASR decoder.
Architecture of the proposed end-to-end EPD framework incorporating AFE, PE, and EOU predictor branches.

To optimize performance, these three networks are not merely concatenated. After independent initial training, they are attached to a final Deep Neural Network (DNN) classifier and undergo Joint Retraining (JRT). This fine-tunes the entire pipeline specifically to minimize endpointing errors rather than optimizing for isolated phonetic or linguistic accuracy.

Probabilistic EOU Prediction

A key differentiator of this model is its move from binary classification to framewise probabilities. Traditional "Word LSTM" approaches attempt to classify words as either turn-taking or not. However, natural language is inherently ambiguous. For example, the word "on" might end the sentence "turn the lights on" or begin the phrase "on the table."

Comparison between (a) conventional binary EOU labels and (b) the proposed framewise EOU probabilities.

By using framewise probabilities, the EOU predictor can handle this ambiguity more gracefully. During the training phase, the system uses a complex ASR module to generate these labels, but this module is discarded during inference. This allows the on-device model to retain the "wisdom" of a large language model while remaining lightweight.

Experimental Results

The framework was evaluated using the CHiME-3 dataset (noisy real-world environments) and the SiTEC Dict01 dataset (large-scale Korean speech). The proposed model consistently outperformed conventional EPD approaches across several key metrics:

  • Reduced Latency: In CHiME-3 tasks, the model achieved a "late endpoint time" of approximately 159ms, significantly lower than the ~198ms observed in standard GLDNN models.
  • Improved Accuracy: By reducing premature cut-offs, the Word Error Rate (WER) improved. On the large-scale Korean task, the proposed model reached a WER of 13.41%, compared to 17.67% for the baseline.
  • Noise Robustness: The ensemble approach proved effective even in unseen noise conditions, demonstrating that the combination of acoustic and linguistic features generalizes well to diverse environments.

Conclusion

This research demonstrates that EPD can be significantly improved by decoupling the training-time linguistic knowledge from the inference-time computational requirements. By utilizing an ensemble of acoustic and language modeling embeddings, the proposed framework offers a high-accuracy, low-latency solution suitable for embedded and mobile ASR applications. Future work may involve integrating more complex attention mechanisms, such as Transformers, to further refine the EOU predictor.

Summary updated

BibTeX
@article{hwang2020end,
  author = {Inyoung Hwang and Joon-Hyuk Chang},
  title = {{End-to-End Speech Endpoint Detection Utilizing Acoustic and Language Modeling Knowledge for Online Low-Latency Speech Recognition}},
  journal = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.3020696},
}
Figure