TL;DR
This paper introduces a novel method to predict the quality of speech document indexing for Spoken Document Retrieval (SDR) systems. By combining Automatic Speech Recognition (ASR) confidence scores with a Latent Dirichlet Allocation (LDA)-based semantic outlier detection, they significantly improve the ability to identify documents that will be well-indexed, outperforming traditional lexical-level approaches.
What Problem Does This Paper Address?
Spoken Document Retrieval (SDR) systems aim to search through large archives of audio content. A crucial first step is speech indexing, which relies on Automatic Speech Recognition (ASR) to transcribe audio into text. However, ASR systems are not perfect and their accuracy can degrade significantly in real-world conditions (e.g., noisy environments, spontaneous speech), leading to errors in the transcriptions. These errors directly impact the effectiveness of SDR, making it difficult to retrieve relevant documents. While ASR confidence measures can offer some indication of transcription quality, they don't account for the semantic meaning of the content or how errors might affect a user's search query. This paper addresses the challenge of predicting the indexing quality of a speech document, going beyond simple transcription accuracy to understand how well a document can be effectively searched.
Methods
The authors propose a two-pronged approach to predict indexing quality:
- ASR Confidence Measure (CM): They leverage existing confidence scores provided by the ASR system for each word. These scores are derived from acoustic, linguistic, and graph features, aiming to predict the probability of a word being correct.
- Latent Dirichlet Allocation (LDA) for Semantic Outlier Detection: This is the core novelty. Instead of just looking at individual word confidence, they use LDA to identify "semantic outliers" – words whose topics are significantly different from the dominant topics within a document. LDA is a probabilistic topic model that represents documents as mixtures of topics and topics as distributions over words. By training LDA on a large corpus (French Wikipedia), they can identify thematic inconsistencies within a speech document's transcription. This approach is more robust than lexical-level methods because it considers the broader semantic context.
These two metrics (CM and LDA-based topic consistency) are then combined using a 3-layer perceptron neural network trained with back-propagation to predict the overall indexing quality (PIdx).
Datasets
- Speech Database: The experiments were conducted on the ESTER2 campaign database, comprising approximately 100 hours of French radio stations and 40 hours of African radio stations. Development and test sets were extracted, each containing 6 hours of radio recordings from 2007-2008. Documents were segmented into segments with a maximum length of 30 seconds.
- French Wikipedia Corpus: Used for training the LDA model to identify thematic classes and compute word topic consistency.
- Query Set: Derived from headlines of "Le Monde" and almanacs (RFI, Le Figaro, French Wikipedia) from 2007-2008, resulting in 63,000 unique queries.
Key Insights and Ideas
- Semantic Outliers Matter: Errors in transcription have a different impact depending on their semantic context. A misrecognized word that is semantically coherent with the rest of the document is less detrimental than a misrecognized word that introduces a completely unrelated topic.
- LDA for Semantic Coherence: LDA provides a powerful way to model the thematic structure of documents and identify words that deviate from this structure, acting as a proxy for semantic errors.
- Topic-Level vs. Lexical-Level: The LDA-based topic-level approach significantly outperforms lexical-level methods (like the Semantic Compacity Index) in predicting indexing quality, validating the importance of semantic context.
- Predicting Indexability is Key: Directly predicting the quality of indexing rather than just the transcription accuracy allows SDR systems to better handle imperfect ASR outputs.
Key Results
- The proposed method, combining ASR confidence and LDA-based topic consistency, significantly reduces prediction error compared to using only ASR confidence or lexical-level semantic measures.
- Specifically, the LDA-based approach (PTEM) reduced prediction error by 6.32% absolute with the Distortion (D) metric and 2.47% with the Root Mean Square (RMS) metric, compared to ASR confidence alone.
- The combination of CM and PTEM provided a slight further improvement.
- In a document classification scenario, the LDA-based method (PTEM) achieved a classification rate of 63% for detecting low-quality documents at a 50% threshold, a significant improvement over ASR confidence (43%).
- The authors report that their method can detect documents impacting ranking (under 40% indexability) and those answering queries correctly (above 70%) with a classification rate generally above 70%.
Limitations and Future Work
- Computational Complexity: LDA can be computationally intensive, especially with large corpora and many topics. The authors mitigate this by computing LDA on smaller parts of the corpus relevant to specific words.
- Topic Genericity: If the number of LDA topics is too low, classes can become too generic, making topic assignment problematic.
- Future Work: The authors plan to directly integrate word-level topic consistency into the confidence measure as a predictor feature, rather than using it as a separate metric. They also aim to explore how semantic or topic information can directly improve the confidence measure itself.
Summary updated