On the relation of Bayes risk, word error, and word posteriors in ASR

Ralf Schlüter, Markus Nußbaum-Thom, Hermann NeyInterspeech 2010

TL;DR

This paper delves into the long-standing inconsistency in Automatic Speech Recognition (ASR) between minimizing sentence-level errors (Bayes decision rule) and the practical evaluation metric of word-level errors. The authors provide analytical evidence and experimental validation showing that, under specific conditions, these two seemingly different objectives lead to the same optimal decisions, offering a clearer understanding of when word-error-minimizing approaches truly offer an advantage.

What Problem Does This Paper Address?

In ASR, the ideal goal is to find the word sequence that minimizes the probability of making any error at all (sentence error). This is typically achieved using the Bayes decision rule, which aims to minimize the "Bayes risk." However, the standard way we evaluate ASR systems is by measuring the Word Error Rate (WER), which focuses on individual word mistakes. This creates a fundamental mismatch: the system is optimized for one thing (sentence error) but judged by another (word error).

Researchers have proposed various methods to align the Bayes decision rule with word error costs, often using Levenshtein distance (which directly models word insertions, deletions, and substitutions). While these approaches seem intuitively better for WER, experiments have often shown only marginal improvements, especially at low error rates. The core problem this paper addresses is understanding why these word-error-minimizing rules don't always yield significant gains and to analytically characterize the conditions under which they do align with the standard sentence-error-minimizing rule.

Methods

The paper starts by formalizing the Bayes decision rule and its relationship to cost functions. The standard approach uses a 0-1 cost function for sentence error, leading to the Maximum A Posteriori (MAP) rule. The authors then introduce the concept of a Levenshtein rule, which uses the Levenshtein distance as the cost function.

A key methodological contribution is the introduction of the "multiple alignment Hamming cost" and "Hamming risk." This is derived by transforming Levenshtein alignments into Hamming alignments by inserting "empty" words. This allows for a more structured analysis of word sequence distances. The Hamming risk is shown to be an upper bound to the Levenshtein risk, providing a way to efficiently approximate or bound the true Levenshtein risk.

Building on this, the paper introduces "Hamming Iteration" (HIT), an iterative process that uses the Hamming risk to refine word sequence hypotheses. This is related to existing "N-best ROVER" (NBR) approaches. The authors then derive analytical conditions under which the MAP, Levenshtein, MWE (Minimum Word Error), NBR, and HIT decision rules all yield identical results.

Datasets

The experiments were conducted on three well-known large vocabulary ASR tasks:

  • Wall Street Journal 5k (WSJ5k): A standard benchmark in ASR research.
  • European Parliament Plenary Sessions (EPPS) English (EPPS-EN): A corpus of parliamentary proceedings.
  • European Parliament Plenary Sessions (EPPS) Spanish (EPPS-ES): The Spanish counterpart to EPPS-EN.

The paper uses N-best lists (lists of the top-k hypothesized word sequences) generated by baseline ASR systems for rescoring and evaluation.

Key Insights and Ideas

  • The "Inconsistency" is Conditional: The paper analytically demonstrates that the perceived inconsistency between sentence-error minimization (MAP) and word-error minimization (Levenshtein) is not absolute. There are specific, quantifiable conditions under which both objectives lead to the same optimal decision.
  • Hamming Risk as a Bridge: The introduction of the Hamming risk provides a computationally tractable way to analyze and approximate Levenshtein-based decision rules. It acts as an upper bound, allowing for efficient exploration of word error minimization.
  • Iterative Refinement (HIT): The Hamming Iteration (HIT) approach offers a practical method for improving ASR hypotheses by iteratively minimizing the Hamming risk. This method is shown to not increase the Levenshtein risk, making it a safe optimization strategy.
  • Low Error Rates Favor Agreement: The most significant insight is that under conditions of low baseline word error rates and/or short average segment lengths, the MAP and word-error-minimizing rules tend to agree. This explains why improvements from word-error-based methods are often marginal in well-performing systems.
  • Posterior Probability is Key: The paper highlights the crucial role of word posterior probabilities. The conditions for agreement between MAP and Levenshtein rules are directly linked to the magnitude of these probabilities and the distribution of probability mass across different word sequences.

Key Results

The experimental results, summarized in Tables 3-5, largely support the analytical findings.

  • For the WSJ5k corpus, a large percentage of segments (over 95%) showed agreement between MAP, MWE, and HIT rules, with WERs around 10.2-10.4%. The segments where they differed ("≠" cases) were small (4.2% of segments) and showed slightly higher WERs (13.2-14.1%).
  • For the EPPS corpora (Spanish and English), which have higher baseline error rates and longer average segment lengths, the proportion of segments where the rules agreed (cases a-c) was smaller, and the "≠" cases were larger, leading to more noticeable differences in WERs between the rules. For EPPS Spanish, the "≠" cases (20.6% of segments) had WERs around 13.8-14.0%, compared to 9.49-9.54% for the agreeing cases.
  • The paper demonstrates that the Hamming Iteration (HIT) consistently achieves WERs close to or equal to the MWE rule, and often better than the MAP rule, particularly in cases where the rules diverge.

Limitations and Future Work

  • Local Optima: The paper acknowledges that while HIT does not increase Levenshtein risk, it is a locally optimal approach and does not guarantee finding the global optimum.
  • Conditions for Agreement: While analytical conditions for agreement are provided, the paper notes that the proof of these conditions for all cases will be presented in a future publication.
  • Future Work: The authors suggest that the insights gained can inform the design of more efficient ASR systems and potentially lead to better system combination strategies. The relationship between word error rate, posterior probabilities, and decision rules could be further explored for more nuanced cost functions.

Summary updated

BibTeX
@inproceedings{schluter2010relation,
  author = {Ralf Schlüter and Markus Nußbaum-Thom and Hermann Ney},
  title = {{On the relation of Bayes risk, word error, and word posteriors in ASR}},
  booktitle = {Interspeech},
  year = {2010},
}

Loading related content...

Figure