A confidence measure invariant to language and grammar

Daniele Colibro, Luciano Fissore, Claudio Vair, Emanuele Dalmasso, Pietro LafaceInterspeech 2005

TL;DR

This paper introduces a novel confidence measure for speech recognition systems that is invariant to language and grammar. This innovation significantly simplifies development by allowing a single rejection threshold to be used across diverse applications and languages, reducing the need for costly, per-application tuning.

What Problem Does This Paper Address?

In speech recognition systems, especially those used in interactive applications like voice assistants, it's crucial to know how confident the system is about its recognition output. This confidence score is used to decide whether to accept a recognized word or sentence or to ask for clarification. A major challenge is that existing confidence measures are often highly dependent on the specific language, grammar, and vocabulary of the application. This means developers have to painstakingly tune rejection thresholds for each individual application, which is time-consuming and expensive. The authors aim to create a confidence measure that behaves consistently across different languages and grammars, making it "application independent."

Methods

The paper proposes a new confidence measure called the "Normalized Differential Confidence" (NDC). It builds upon an earlier "Acoustic Log Likelihood Ratio" (ALLR) measure. The core idea is to leverage the posterior probabilities of acoustic-phonetic units generated by the system's hybrid HMM-ANN model.

Here's a breakdown of the key steps:

  1. Original Confidence Measure (ALLR): This measure compares the sum of frame scores constrained by a hypothesized word model against the "free score" (the best possible score for each frame). It's a ratio that ranges from 0 to 1, with higher values indicating better acoustic matching.
  2. Differential Confidence (DC): The authors first propose a variant that calculates the difference between the constrained and unconstrained posterior probabilities for each frame, then averages these differences. This measure is sensitive to acoustic mismatches.
  3. Normalization: The crucial step is normalizing the differential confidence. They collect statistics (PDFs and CDFs) of the differences for each acoustic state. By applying the cumulative distribution function (CDF) to these differences, they transform the random variables into a uniformly distributed range [0-1]. This normalization process aims to make the confidence measure invariant to the specific characteristics of each acoustic state.
  4. Linear Compensation: Finally, a simple linear transformation (y = a * NDC + β) is applied to further adjust the NDC values, aiming to achieve a consistent average rejection rate across different languages and grammars.

Datasets

The paper uses existing built-in grammars from the Loquendo ASR system for evaluation. Specifically, they tested with six grammars in UK-English and six in Spanish. No new datasets were introduced.

Key Insights and Ideas

  • The "Normalization" Goal: The primary insight is that by normalizing the confidence measure using the cumulative distribution functions of the differences between constrained and unconstrained acoustic probabilities, the measure becomes significantly less sensitive to variations across languages and grammars.
  • Uniformity is Key: The goal is to achieve rejection functions that are clustered together, meaning a single threshold yields similar rejection rates for different grammars and languages. This uniformity is the hallmark of an "application independent" measure.
  • CDF as a Normalizer: Using the Cumulative Distribution Function (CDF) is highlighted as an effective way to transform random variables into a uniform distribution, which is essential for creating a homogeneous confidence measure.
  • Practical Implications: The ability to use a single rejection threshold dramatically simplifies the development and deployment of speech recognition applications, especially for user-defined grammars where individual tuning is infeasible.

Key Results

The paper demonstrates that the new Normalized Differential Confidence (NDC) measure leads to significantly more homogeneous rejection functions compared to the previous ALLR measure. Figures 7 and 8 (for UK-English and Spanish, respectively) show that the family of curves for in-grammar utterances are much more tightly clustered than those in Figures 1 and 2. This clustering allows for the preset of uniform rejection thresholds, achieving comparable rejection behaviors across different grammars and languages. The authors state that this approach has been successfully applied to all 15 languages supported by the Loquendo ASR system.

Limitations and Future Work

  • Noise Regions: The authors acknowledge that the current approach does not reliably identify noise regions in sentences. In these regions, both constrained and unconstrained probabilities are similar and good, which can introduce a bias into the total confidence score. Eliminating this bias is identified as a future research direction.
  • Statistical Independence Assumptions: While the NDC measure is designed to be homogeneous, the authors note that the statistical independence and homogeneity assumptions for the contributing terms are not perfectly met, leading to a mean value slightly greater than 0.5 for the NDC distribution (which is expected to be Gaussian with mean 0.5).

Summary updated

BibTeX
@inproceedings{colibro2005confidence,
  author = {Daniele Colibro and Luciano Fissore and Claudio Vair and Emanuele Dalmasso and Pietro Laface},
  title = {{A confidence measure invariant to language and grammar}},
  booktitle = {Interspeech},
  year = {2005},
}
Figure