TL;DR
This paper introduces a novel dynamic language model adaptation technique for spoken keyword recognition in information retrieval tasks. By classifying keyword categories and their positions within a query utterance, the system adaptively adjusts language model weights, leading to a significant 22.0% reduction in keyword error rates.
What Problem Does This Paper Address?
Spoken information retrieval (IR) systems, like those used to find TV programs, often need to understand user queries spoken in spontaneous, unconstrained speech. A key challenge is accurately identifying and classifying keywords (e.g., names, places, times) within these queries, especially when multiple keywords from different categories are present in a single short utterance. Traditional language models struggle with this dynamic nature, requiring lengthy contexts or being too rigid. This paper tackles the problem of adapting language models on-the-fly to better recognize keywords in short, spontaneous spoken queries, even when keyword categories vary within the query.
Methods
The proposed method employs a two-pass approach:
- First Pass Recognition & Keyword Category Classification:
- A standard Large Vocabulary Continuous Speech Recognition (LVCSR) system processes the spoken query to generate an initial word sequence.
- Conditional Random Fields (CRF) are then used to classify each word in this sequence into a specific keyword category (e.g., "person name," "program name," "channel name," or "non-keyword"). CRFs are chosen for their ability to model sequential data and incorporate rich, discriminative features.
- Crucially, the CRF utilizes long-distance linguistic features by considering word co-occurrences within the utterance, going beyond simple local n-gram history. Features include word IDs, confidence scores, syllable counts, and preceding pause information. This allows for more robust classification, even in the presence of speech recognition errors or disfluencies.
- The output of the CRF is the posterior probability of each keyword category for each word.
- Second Pass Recognition with Dynamic Adaptation:
- The system then re-processes the spoken query using a dynamically adapted language model.
- This adapted model is a mixture of individual language models, each trained for a specific keyword category.
- The mixture weights for these category-specific models are determined by the posterior probabilities estimated by the CRF in the first pass. These weights vary dynamically across the utterance, reflecting the changing keyword categories. This allows the system to prioritize the language model most relevant to the current segment of the query.
Datasets
- Training Data: A collection of 27,431 query sentences for a television program retrieval task was used to train the baseline trigram language model and the "non-keyword" language model.
- Evaluation Data: 28,845 utterances from 210 users were collected for a television program retrieval task. A test set of 5,841 utterances from 40 speakers, containing 2,107 keywords, was used for evaluation. The data was segmented into four categories: "person name," "program name," "channel name," and "non-keyword."
- CRF Training Data: Sentences containing at least one keyword from the collected data were used to train the CRF.
Key Insights and Ideas
- Leveraging Long-Distance Linguistic Features: The use of CRFs with features that capture word co-occurrences across longer distances within an utterance is key to improving keyword category classification accuracy, especially in spontaneous speech. This moves beyond traditional local context modeling.
- Dynamic Weighting for Category-Specific LMs: The core innovation is the dynamic adjustment of mixture weights for category-specific language models based on real-time keyword category classification. This allows the system to adapt to the changing linguistic context of a query on a frame-by-frame basis.
- Robustness to Recognition Errors: The method shows promise in recovering from initial speech recognition errors because the CRF can still infer keyword categories even if words are misrecognized, provided sufficient contextual features are available.
- Effectiveness in Multi-Keyword Queries: The proposed method significantly outperforms baseline models when queries contain multiple keywords (N ≥ 2), demonstrating its ability to handle complex query structures.
Key Results
- Keyword Category Classification: The CRF achieved an average classification rate of 57.2% for keyword categories and their positions, and 42.2% even when keywords were mis-recognized in the first pass.
- Keyword Error Rate Reduction: The proposed dynamic adaptation method achieved a 22.0% reduction in keyword error rate compared to the baseline non-adapted model for utterances with one or more keywords.
- Performance with Multiple Keywords: For queries with two or more keywords (N ≥ 2), the keyword error rate was reduced by 20.8% compared to the baseline.
- Oracle Performance: The proposed method achieved performance comparable to an "oracle" condition where the correct keyword categories were known beforehand.
Limitations and Future Work
- Non-Keyword Insertion Errors: The paper notes that keyword insertion errors were more numerous for utterances with no keywords (N=0). This is attributed to the CRF's training on sentences with at least one keyword, potentially leading it to misclassify non-keywords.
- Ineffectiveness in Complete Recognition Failure: For utterances where the initial speech recognition was a near-total failure, the method's effectiveness was limited due to the lack of usable co-occurrence features.
- Future Work:
- Improve keyword category classification for segments containing recognition errors.
- Explore methods to filter out utterances that likely do not contain keywords or have low confidence scores.
- Evaluate the impact of the proposed method on the overall information retrieval performance, not just speech recognition accuracy.
Summary updated