N-best speech hypotheses reordering using linear regression

Ananlada Chotimongkol, Alexander I. RudnickyInterspeech 2001 · OpenReview

TL;DR

This paper tackles the problem of improving speech recognition accuracy in dialog systems by reordering the "n-best" hypotheses generated by the recognizer. They propose a linear regression model that combines external linguistic features (like syntactic parses and dialog context) with acoustic features to better predict the correct utterance, outperforming human performance on the same task.

What Problem Does This Paper Address?

Speech recognition systems often generate a list of the most likely transcriptions (n-best hypotheses) for an utterance, rather than just a single best guess. However, the top-ranked hypothesis isn't always the most accurate. In dialog systems, this inaccuracy can lead to misunderstandings and task failures. The challenge is to effectively leverage additional information beyond the raw acoustic signal and the recognizer's initial scoring to reorder these hypotheses and bring the truly correct one to the top. This paper focuses on using features derived from the linguistic structure of the utterance and its dialog context to achieve this reordering.

Methods

The authors propose a post-processing approach to reorder n-best speech hypotheses. This means the reordering happens after the initial speech recognition process, making it less complex than integrating reordering into the core recognition engine.

The core of their method is a linear regression model. This model takes various features as input and predicts a "correctness score" for each hypothesis. The hypothesis with the highest predicted correctness score is then chosen as the reordered top hypothesis.

The key innovation lies in the features used:

  • Syntactic Features: These leverage the output of a parser (specifically, the Phoenix parser, designed for spontaneous speech) to assess how well a hypothesis conforms to the expected grammar of the air travel domain. Two specific features are used:
  • Coverage: The proportion of words in the hypothesis that are successfully parsed.
  • Fragmentation: The number of switches between parsed and unparsed segments in the parse output.
  • Semantic Features:
    • Correlation between system and user utterances: This captures the likelihood of a user utterance given the current dialog state and topic slots.
  • Relationship between topic slots: This models dependencies between different pieces of information (slots) within a user's utterance.
  • Acoustic Features: While the recognizer already uses these, the authors suggest they can still be valuable for reordering, especially for distinguishing between similar hypotheses or for utterances like simple yes/no answers.

A novel aspect is the introduction of Concept Error Rate (CER) as an alternative evaluation metric to the standard Word Error Rate (WER). CER focuses on the semantic meaning (concepts represented by slots and their values) rather than the exact surface words, making it more relevant for dialog system performance where semantic understanding is crucial.

Datasets

The paper uses data from the Communicator system, a dialog system for travel planning.

  • Training Data: The linear regression model was trained on approximately 35,776 utterances from the Communicator data collected between June 1998 and May 1999.
  • Test Data: The model was evaluated on 1,718 utterances from the June 1999 Communicator test set.

Key Insights and Ideas

  • Linguistic context is powerful for reordering: Features derived from syntactic parses and dialog context significantly improve hypothesis selection beyond what acoustic features alone can achieve.
  • Concept Error Rate (CER) is a better metric for dialog systems: CER aligns better with human judgment of dialog success because it focuses on semantic meaning, tolerating variations in wording and non-essential words that don't affect task completion.
  • Human performance is a strong benchmark: The proposed linear regression model achieves performance comparable to human subjects in reordering hypotheses, suggesting that automatic reordering can be highly effective.
  • N-best list size matters: The accuracy of the reordering model degrades as the hypothesis list size increases beyond a certain point (top-5 in their experiments), likely due to increased noise at lower ranks.

Key Results

  • The linear regression model achieved a 11.97% relative improvement in Word Error Rate (WER) compared to the baseline recognizer.
  • It also achieved a 27.61% relative improvement in Concept Error Rate (CER).
  • The proposed model performed as well as human subjects in selecting the best hypothesis from the n-best list.
  • When native speakers were allowed to arbitrarily edit hypotheses, they achieved even better error rate reductions than the "oracle" (perfect knowledge), suggesting valuable information exists outside the n-best list.

Limitations and Future Work

  • Limited improvement over some prior work: The authors acknowledge that their WER improvement is less than some other methods reported on different corpora (like ATIS). They hypothesize this might be due to shorter utterances in their test set, which may not provide enough structure for parse-level features to be effective.
  • Future Work:
    • Incorporate more word-level features: Such as word confidence scores and word frequency within the n-best hypotheses, as these are commonly used by humans.
  • Optimize for Concept Error Rate: Directly predict correctness scores based on CER, while accounting for its skewed distribution.
  • Extract more human-like features: Investigate features related to syntax, topic conformance, insertions/deletions, and response naturalness, which human subjects reported using.

Summary updated

Abstract

We propose a hypothesis reordering technique to improve speech recognition accuracy in a dialog system. For such systems, additional information external to the decoding process itself is available, in particular features derived from the parse and the dialog. Such features can be combined with recognizer features by means of a linear regression model to predict the most likely entry in the hypothesis list. We introduce the use of concept error rate as an alternative accuracy measurement and compare it withy the use of word error rate. The proposed model performs better than human subjects performing the same hypothesis reordering task.
BibTeX
@inproceedings{chotimongkol2001n,
  author = {Ananlada Chotimongkol and Alexander I. Rudnicky},
  title = {{N-best speech hypotheses reordering using linear regression}},
  booktitle = {Interspeech},
  year = {2001},
  url = {https://openreview.net/forum?id=E0SP3BEaBc},
}
Figure