Robust speech recognition using VAD-measure-embedded decoder

Tasuku Oonishi, Paul R. Dixon, Koji Iwano, Sadaoki FuruiInterspeech 2009

TL;DR

This paper introduces a novel approach to Voice Activity Detection (VAD) that embeds speech/non-speech confidence scores directly into the speech recognition decoder. This allows the decoder to dynamically adjust acoustic model scores based on the confidence of speech presence, leading to significant improvements in word accuracy and a reduction in computational cost, especially in noisy environments.

What Problem Does This Paper Address?

Speech recognition systems often struggle with noisy environments and long periods of silence or non-speech sounds. Traditional Voice Activity Detectors (VADs) typically perform a hard decision: classifying a frame as either speech or non-speech and discarding non-speech frames. This hard decision discards valuable information about the confidence of the classification. When noisy frames are misclassified as speech, they can lead to insertion errors in the recognition output. The paper addresses the challenge of leveraging the degree of confidence in speech/non-speech classification to improve recognition accuracy and efficiency, rather than relying on a binary decision.

Methods

The core innovation lies in embedding the VAD's confidence measure into the speech recognition decoder. Instead of simply discarding non-speech frames, the system uses a continuous Speech/Non-Speech (SNS) score.

  1. Confidence Measure Calculation: The paper uses Gaussian Mixture Models (GMMs) to model speech and non-speech characteristics. For each frame, it calculates likelihoods for speech ($p(X^i|H_1)$) and non-speech ($p(X^i|H_0)$). These are then converted into confidence measures: $C_{H_1}$ (confidence of speech) and $C_{H_0}$ (confidence of non-speech), which sum to 1.
  2. Acoustic Score Biasing:
    • If a hypothesis belongs to a phonetic (speech) model, its acoustic score is biased by adding a scaled version of $C_{H_1}$.
  3. If a hypothesis belongs to a silence model, its acoustic score is biased by adding a scaled version of $C_{H_0}$.
  4. A smoothing parameter ($n$) is used to average these confidence measures over a window of frames, introducing a controlled latency.
  5. Combination with Front-end VAD: The proposed method can be combined with traditional front-end VADs. This allows for early rejection of frames with very high confidence of being non-speech, significantly reducing computational load without sacrificing accuracy.

Datasets

  • Acoustic Model Training: Corpus of Spontaneous Japanese (CSJ) and Japanese Newspaper Article Sentences (JNAS) corpus.
  • Evaluation: Drivers Japanese Speech Corpus in a Car Environment (DJSC). This corpus features command-and-control utterances recorded in a car, with natural non-speech regions and background noise.

Key Insights and Ideas

  • Continuous Confidence is Key: Treating VAD output as a continuous confidence score rather than a binary decision provides richer information that the decoder can exploit.
  • Biasing for Robustness: Dynamically biasing acoustic model scores based on speech confidence helps the decoder to better discriminate between actual speech and noise, reducing insertion errors.
  • Synergy with Front-end VAD: Combining the proposed decoder-based confidence biasing with traditional frame-dropping front-end VADs offers a powerful way to achieve both high accuracy and significant computational savings.
  • Trade-off Management: The smoothing parameter ($n$) and scaling factor ($\alpha$) allow for tuning the trade-off between accuracy, latency, and computational cost.

Key Results

  • Word Accuracy Improvement: The proposed method achieved a significant improvement in word accuracy, reaching 53.1% compared to a baseline without VAD (43.1%), ZCR-based VAD (46.5%), and GMM-based VAD (45.8%). This represents an absolute improvement of 10% over the non-VAD baseline.
  • Computational Cost Reduction: When combined with a front-end GMM-based VAD, the system could reduce computational cost by up to 60% by dropping frames with high confidence of non-speech, while maintaining the highest word accuracy.
  • Parameter Sensitivity: The paper shows that accuracy improves with increased smoothing (larger $n$), and the system becomes more robust to parameter choices. The best performance was achieved with a smoothing frame of 15 and a scaling factor $\alpha$ of 10.

Limitations and Future Work

  • Parameter Tuning: The current parameter settings (e.g., $\alpha$, $n$) were optimized manually. Future work aims to investigate automatic on-line parameter adjustment.
  • Robustness in Diverse Conditions: Further evaluations are planned to assess the robustness of the proposed method across a wider range of noise and Signal-to-Noise Ratio (SNR) conditions.
  • Generalizability: The authors aim to ascertain if the observed performance gains are generalizable and if on-line parameter adjustments are crucial for fully leveraging the method in varying environmental conditions.

Summary updated

BibTeX
@inproceedings{oonishi2009robust,
  author = {Tasuku Oonishi and Paul R. Dixon and Koji Iwano and Sadaoki Furui},
  title = {{Robust speech recognition using VAD-measure-embedded decoder}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure