A SEGMENT-LEVEL CONFIDENCE MEASURE FOR SPOKEN DOCUMENT RETRIEVAL

Grégory Senay, Georges Linarès, François PortetHAL 2011

TL;DR

This paper introduces a novel method to predict the "indexability" of spoken document segments for retrieval systems. By combining acoustic and linguistic confidence scores from Automatic Speech Recognition (ASR) with a new "semantic compacity index," they significantly improve the ability to identify segments that are likely to be relevant and accurately transcribed, a crucial step for efficient spoken document retrieval.

What Problem Does This Paper Address?

Spoken Document Retrieval (SDR) systems aim to find information within audio or video recordings. A major hurdle for SDR is the inherent inaccuracy of Automatic Speech Recognition (ASR) systems. Even with advancements, ASR can produce a significant number of errors, especially in challenging conditions, which can severely degrade the performance of search engines. This paper tackles the problem of predicting how much these ASR errors will impact the overall SDR performance for a given segment of speech. The goal is to develop a self-diagnostic tool that can help identify segments that are likely to be problematic for retrieval, allowing for targeted human correction and improving the efficiency of the indexing process.

Methods

The core of the proposed method is a two-pronged approach to predict the "indexability" of a speech segment.

  1. ASR Confidence Measure (CM): This is a multi-stage process. First, low-level acoustic features (like log-likelihoods) and high-level linguistic features (from a 3-gram language model) are extracted for each word. These features are then fed into a boosting algorithm classifier, trained on a separate corpus where words are manually tagged as correct or erroneous. This yields a word-level confidence score. For a segment, this is computed by averaging the scores of its words after filtering out common "stop words."
  2. Semantic Compacity Index (SCI): This is a novel contribution. The authors hypothesize that semantic relevance is key for SDR, and ASR confidence alone doesn't capture this. SCI estimates how semantically coherent a segment is by looking at short-term correlations between meaningful words. They use lemmatized word pairs and their co-occurrence frequencies, weighted by TF-IDF, to calculate a semantic score for each pair. These scores are then averaged within a sliding window to get a segment-level SCI.
  3. Combination: A 3-layer Multi-Layer Perceptron (MLP) is used to combine the ASR Confidence Measure (CM) and the Semantic Compacity Index (SCI) to predict the final indexability score. The MLP is trained on a portion of the ESTER test set.

Datasets

  • ESTER Database: This is the primary dataset used, consisting of approximately 100 hours of French radio broadcasts. For experiments, 8 hours of test data from 4 different radio stations were used. A 1-hour subset was used for training the predictor, and the remaining 7 hours (1474 documents) were used for testing.
  • French Newspaper "Le Monde" Corpus: Used for training the language models for the ASR system.
  • Wikipedia Corpus (French part): Used for computing the statistics for the Semantic Compacity Index.

Key Insights and Ideas

  • Semantic Information is Crucial for SDR: While ASR confidence measures are useful, they don't fully capture the nuances of how well a segment will perform in a retrieval task. Semantic coherence plays a significant role.
  • Complementarity of Features: The combination of ASR confidence and semantic compacity yields significantly better results than either feature alone. This highlights that different aspects of speech quality and content contribute to retrieval performance.
  • Predicting Indexability for Human Correction: The proposed method is particularly valuable for semi-automatic SDR scenarios where human operators need to identify segments for correction. The indexability score can guide this process efficiently.
  • Adaptable Thresholds for Trade-offs: The indexability predictor allows for flexible thresholding, enabling a trade-off between the cost of human correction and the quality of the indexed documents.

Key Results

  • Prediction Error Rates (PER): The combined CM+SCI approach significantly outperformed CM-only and SCI-only predictions. The Q1 (general distortion) was reduced from 1.65 (CM) and 1.74 (SCI) to 1.59 (CM+SCI). The RMS error was reduced from 0.25 (CM) and 0.32 (SCI) to 0.22 (CM+SCI).
  • Document Classification: When used to classify documents as indexable or unindexable, the CM+SCI predictor achieved a classification rate of 74% at a threshold of 70%, a substantial improvement over CM (62%) and SCI (61%) alone. The combined system correctly tagged over 70% of speech segments in most cases.

Limitations and Future Work

  • Local Semantic Compacity: The current semantic modeling focuses on short-term correlations. The authors suggest exploring context widening and other modeling paradigms like Latent Dirichlet Allocation (LDA) to improve the extraction and identification of latent concepts.
  • ASR System Specificity: The confidence measures are derived from a specific ASR system. Further research could explore the generalizability of the approach to different ASR systems.
  • Dataset Specificity: While ESTER is a good benchmark, testing on diverse datasets with varying acoustic conditions and languages would be beneficial.

Summary updated

Abstract

International audience
BibTeX
@inproceedings{senay2011segment,
  author = {Grégory Senay and Georges Linarès and François Portet},
  title = {{A SEGMENT-LEVEL CONFIDENCE MEASURE FOR SPOKEN DOCUMENT RETRIEVAL}},
  booktitle = {HAL},
  year = {2011},
}
Figure