Introduction
In voice-controlled systems such as Alexa or Siri, "endpointing"—the task of deciding exactly when a user has finished speaking—is a critical component of the user experience. This task presents a fundamental trade-off: an overly aggressive system may cut the user off mid-sentence (high "early endpoint" rate), while an overly cautious system creates an awkward, unnatural silence before responding (high latency).
The primary challenge is the ambiguity of silence. A pause can represent a mid-sentence hesitation or the definitive end of a query. Traditional Voice Activity Detectors (VAD) typically rely on audio energy or simple acoustic features, which lack the linguistic context necessary to distinguish between these two states.
Methodology: Beyond the 1-Best Hypothesis
While some modern systems integrate the speech recognizer's decoder to improve endpointing, they often rely on the single highest-scoring (1-best) hypothesis. This approach can be unstable, as the 1-best guess often "flip-flops" during live decoding.
This paper introduces a novel algorithm that calculates the expected pause duration across the entire search beam. By integrating acoustic and language model (LM) data from all active decoding hypotheses, the system produces a smoother, more reliable signal for decision-making.
Key Components
- Expected Pause Duration ($D(L_t)$): A probability-weighted average of pause durations across all active hypotheses.
- Expected End-of-Sentence Pause ($D_{end}(L_t)$): A specific metric tracking hypotheses that the language model identifies as valid stopping points.
- Decision Logic: An endpoint is triggered if the expected end-of-sentence pause exceeds a specific threshold, or if the overall expected pause becomes extremely long (serving as a safety net for LM failures).
Evaluation and Results
The proposed method was evaluated using an internally collected Amazon dataset, with ground truth established via forced-alignment. The system was compared against a standard energy-based VAD and a 1-best decoder-integrated approach.
As shown in the table below, the proposed "expected-pau" method achieved significant improvements across all key metrics relative to the energy-VAD baseline:
- Accuracy: It reduced the Early Endpoint Rate (EEPR) by 45% and the Missed Endpoint Rate (MEPR) by 43%.
- Word Error Rate (WER): It achieved a 5% relative reduction in WER, likely by preventing the premature truncation of words at the end of utterances.
- Latency: Crucially, these accuracy gains were achieved while maintaining a latency nearly identical to the simple energy-VAD (1.02 relative), whereas the 1-best approach was significantly slower (1.32 relative).
Conclusion
By shifting from a single-hypothesis view to an expectation over the entire decoding beam, this approach provides a robust mechanism for linguistically-aware endpointing. The system "knows" when a sentence is grammatically incomplete, allowing it to tolerate hesitations without sacrificing response speed. Future work may explore more integrated methods for handling the initial silence at the start of an audio stream and further optimizing the computational cost of beam-wide expectations.
Summary updated