Score Distribution Based Term Specific Thresholding for Spoken Term Detection

Doğan Can, Murat SaraçlarNAACL 2009

TL;DR

This paper introduces a novel "Term Specific Thresholding" (TST) method for Spoken Term Detection (STD) that leverages per-query posterior score distributions. By modeling these distributions with a mixture of exponential functions, the approach achieves higher maximum precision, crucial for applications like sign language dictionaries where accuracy is paramount.

What Problem Does This Paper Address?

Searching through vast multimedia archives, especially spoken content, is a significant challenge. Spoken Term Detection (STD) aims to pinpoint specific words or phrases within audio recordings. A key hurdle in STD is the inherent error rate of Large Vocabulary Continuous Speech Recognition (LVCSR) systems, which can lead to missed detections or false alarms. While traditional methods use a single, global threshold to decide if a detected term is a true match, this paper addresses the need for more nuanced decision-making, particularly in scenarios where high precision is critical, even at the expense of recall. The authors focus on applications like creating sign language dictionaries for the hearing impaired, where returning incorrect sign information is highly undesirable.

Methods

The paper builds upon a standard STD pipeline that first generates word-level lattices from an LVCSR system. These lattices, which represent multiple possible transcriptions with associated probabilities, are then indexed using Weighted Finite State Transducers (WFSTs). Retrieval involves querying this index to obtain a list of candidate term occurrences ranked by their posterior probabilities.

The core innovation lies in the decision mechanism. Instead of a single global threshold, the authors propose Term Specific Thresholding (TST). This method models the distribution of posterior scores for each query separately. They use a mixture of two exponential distributions to represent the scores, distinguishing between correct and incorrect detections. This mixture model's parameters are learned using an Expectation-Maximization (EM) algorithm. Once these distributions are estimated, a Minimum Bayes Risk (MBR) threshold is calculated for each query. This threshold is optimized to minimize a cost function that balances the cost of missing a correct term (miss) against the cost of a false alarm. The parameter 'α' in their formulation controls this trade-off, allowing for flexible decision-making based on application needs.

Datasets

The experiments were conducted on 11 hours of test material from Turkish Radio and Television Channel 2 (TRT2) broadcasts, specifically news programs for the hearing impaired. This data was used to extract 10,229 single-word queries from reference transcriptions. The ASR system was trained on a larger dataset comprising 100 hours of speech and transcription data from various sources, including TRT2, other broadcasts, and a general text corpus of 100 million words.

Key Insights and Ideas

  • Query-Specific Nuance is Key: A single global threshold is insufficient for optimizing performance across all queries, especially when precision is paramount. Tailoring the decision threshold to the specific characteristics of a query's score distribution can yield significant improvements.
  • Score Distributions Reveal More: The posterior scores returned by STD systems are not uniformly distributed. Modeling these distributions, particularly the separation between correct and incorrect detections, provides a richer signal for making accurate decisions.
  • Exponential Mixtures are Effective: The paper demonstrates that a mixture of exponential distributions is a suitable parametric model for capturing the posterior score distributions in STD, allowing for robust estimation via the EM algorithm.
  • Precision Gains in High-Precision Region: The proposed TST method excels in the high-precision region of the precision-recall curve, directly addressing the needs of applications where false alarms are highly detrimental.

Key Results

The proposed Term Specific Thresholding (TST) method, using score distribution modeling and Minimum Bayes Risk, significantly outperforms Global Thresholding (GT). It also shows advantages over the Term-Weighted Value (TWV) based term-specific thresholding, particularly in achieving higher maximum precision. While TWV-TST performs well, the proposed method achieves precision values beyond the 96.5% limit observed for GT and TWV-TST. The paper presents a precision-recall curve (Figure 2) illustrating these gains, with the proposed method consistently achieving higher precision for a given recall level, especially at high precision values.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Initial Parameter Estimation: The current implementation relies on loose assumptions for initial parameters of the likelihood functions and uses uninformative prior distributions. Better initial estimates, potentially derived from training data, could further improve performance.
  • Parametric Form of Likelihood: The assumption of an exponential distribution for likelihood functions might not always be valid. Exploring other parametric forms could lead to better modeling of posterior score distributions.
  • Overtraining with Limited Data: With limited data, especially when few results are returned for many queries, maximum likelihood estimation can be prone to overtraining.
  • Robustness with Bayesian Methods: The authors suggest that Bayesian methods could be employed to introduce priors on model parameters, making the estimation more robust, especially in data-scarce situations.
  • Supervised Estimation: The paper presents an "upper bound" curve achieved by using correct class labels for parameter estimation in a supervised manner. This highlights the potential for improvement if more accurate supervised information were available.

Summary updated

BibTeX
@inproceedings{can2009score,
  author = {Doğan Can and Murat Saraçlar},
  title = {{Score Distribution Based Term Specific Thresholding for Spoken Term Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
}
Figure