Confidence measures for spontaneous speech recognition

T. Schaaf, T. Kemp1997 IEEE International Conference on Acoustics, Speech, and Signal Processing · DOI

TL;DR

The authors introduce JANKA, a measure-of-confidence (MOC) tagger designed to predict word-level errors in spontaneous speech recognition. By combining acoustic, linguistic, and search-graph features into a neural network classifier, JANKA achieves a tagging accuracy of 90% on spontaneous German dialogs, providing a reliable signal for downstream tasks like unsupervised model adaptation and dialogue repair.

What Problem Does This Paper Address?

Even state-of-the-art Automatic Speech Recognition (ASR) systems are prone to errors, particularly with spontaneous, human-to-human speech. The critical issue is that these systems are often "blind" to their own mistakes; they produce a hypothesis without indicating which words are likely correct and which are guesses.

If a system could reliably flag low-confidence words, that metadata could be used to:

  1. Improve Adaptation: Select only high-confidence words for unsupervised speaker adaptation (e.g., MLLR).
  2. Multi-modal Fusion: Weight non-speech sources (like lip-reading) more heavily when audio confidence is low.
  3. Dialogue Management: Trigger "repair" sub-dialogues in Natural Language Processing (NLP) systems when a semantically vital word is flagged as uncertain.

Methods

The researchers developed JANKA using a two-stage pipeline: feature engineering and classification.

1. Feature Engineering

They started with 19 candidate features, including:

  • A-stabil: The most powerful feature, measuring how often a word appears in a set of alternative hypotheses generated by varying the weights between acoustic and language models.
  • Search Graph Statistics: Features like LogAWE-end (number of active final word states) and N-active-leafs.
  • Acoustic/LM Scores: Normalized acoustic scores (NScoreQ) and language model backoff behavior (LM-NGRAM).
  • Prosodic/Temporal Features: Speaking rate (SpkRate) and signal-to-noise ratio (SNR).

2. Feature Selection

The authors calculated correlation coefficients between these features and the actual correctness of the words (see Table 2 below). While most features showed linear dependencies, SpkRate and A-Entropy were kept despite low linear correlation because they exhibited significant non-linear relationships with error rates.

Correlation coefficients of candidate features with word correctness.

3. Classification

They compared two approaches:

  • Linear Discriminant Analysis (LDA): A simple linear classifier that projects features into a one-dimensional space.
  • Neural Network (NN): A 3-layer multilayer perceptron. Interestingly, they found that a very simple NN with shortcut connections and a single hidden unit performed best. They also tested a version that included the feature vectors of neighboring words to provide context.

Datasets

The study utilized the GSST database, collected under the VERBMOBIL project.

  • Content: Spontaneous German human-to-human dialogs focused on appointment scheduling.
  • Scale: Approximately 33 hours of speech.
  • Evaluation Set: A specific subset of 1,251 utterances (roughly 162 minutes) from speakers not included in the main training database was used to train and test the MOC tagger.
Composition of the additional database used for MOC training and evaluation.

Key Insights and Ideas

  • Stability is the Best Signal: The A-stabil feature (stability across different model weightings) had the highest correlation (0.481) with word correctness. If a word persists even when you tweak the "rules" of the recognizer, it is highly likely to be correct.
  • Non-Linearity Matters: The relationship between error rate and speaking rate is U-shaped; errors increase significantly when speech is either unusually slow or unusually fast. This highlights why simple linear correlation isn't always enough for feature selection.
  • Contextual Gains: Errors in ASR often cluster. By feeding the neural network the features of neighboring words ("NN with context"), the authors achieved their best results, suggesting that the local environment of a word provides vital clues about its reliability.
  • Normalization Strategy: Normalizing acoustic scores using a-priori frame probabilities was found to be more effective than using scores from a dedicated phone recognizer.

Key Results

The baseline word accuracy of the JANUS-3 recognizer on this difficult spontaneous task was 82%. If the system simply tagged every word as "correct," it would achieve a baseline tagging accuracy of 85.3%.

JANKA significantly outperformed this baseline:

  • Linear Classifier: 89.3% accuracy (27.2% error reduction).
  • Neural Network with Context: 90.0% accuracy (32.0% error reduction).
  • High Precision: At a recall rate of 84% (spotting 84% of all correct words), JANKA achieved a precision of over 95%. This high precision is critical for tasks like unsupervised adaptation, where you cannot afford to train on incorrect data.
Precision and Recall curves for the Neural Network classifier.

Limitations and Future Work

  • System Specificity: The features and classifier were tuned specifically for the JANUS-3-SR recognizer. While the principles are generalizable, the specific weights and feature utility might vary for other architectures (like modern End-to-End Transformers).
  • Domain Constraints: The experiments were conducted within the appointment scheduling domain. Performance might degrade in more open-ended or noisy environments.
  • Future Applications: The authors suggest that the next logical step is integrating JANKA into an unsupervised adaptation loop to see if the high-precision confidence tags lead to measurable improvements in baseline ASR accuracy over time.

Summary updated

BibTeX
@inproceedings{schaafconfidence,
  author = {T. Schaaf and T. Kemp},
  title = {{Confidence measures for spontaneous speech recognition}},
  booktitle = {1997 IEEE International Conference on Acoustics, Speech, and Signal Processing},
  doi = {10.1109/icassp.1997.596075},
}
Figure