Improved spoken term detection by discriminative training of acoustic models based on user relevance feedback

Hung-yi Lee, Chia-Ping Chen, Ching-feng Yeh, Lin-Shan LeeInterspeech 2010 · OpenReview

Introduction

Spoken Term Detection (STD) involves searching for specific words or phrases within large audio archives. While traditional systems perform well on clean, supervised data, their performance degrades significantly in real-world scenarios—such as noisy internet videos or spontaneous lectures—where there is a mismatch between the training and testing environments.

Standard acoustic model training focuses on minimizing Word Error Rate (WER) to improve transcription accuracy. However, for search tasks, perfect transcription is often unnecessary. Instead, the primary goal is to rank relevant audio segments higher than irrelevant ones. This paper proposes a novel framework that bridges the gap between speech recognition and information retrieval by using user relevance feedback to discriminatively retrain acoustic models.

System Architecture

The authors introduce a "closed-loop" system designed to refine search results iteratively. In this framework, the retrieval engine initially provides a list of results based on a user query. The user then provides feedback by marking segments as either relevant (True) or irrelevant (False). This feedback is fed back into the system to adapt the acoustic models, which are then used to re-decode the audio and rerank the results.

The proposed closed-loop framework for spoken term detection using user relevance feedback.

Methodology

The framework operates in two primary scenarios:

  1. Short-term Context Feedback: The system performs an initial search, the user labels the top $N$ results, and the system immediately retrains and reranks the remaining results for that specific query.
  2. Long-term Context Feedback: Feedback is collected across multiple queries over time to create a globally improved acoustic model for future searches.

Discriminative Objective Functions

To optimize for retrieval rather than just recognition, the authors developed four objective functions ($F_1$ through $F_4$):

  • $F_1$ & $F_2$: Aim to maximize the scores of positive examples or the margin between positive and negative examples.
  • $F_3$ (Ranking-based): Updates the model only when a relevant segment is ranked lower than an irrelevant one, directly targeting retrieval performance.
  • $F_4$ (Smoothing): Incorporates a smoothing term using unlabeled data to prevent overfitting to the typically small set of user-provided labels.

Optimization is achieved using a "weak-sense auxiliary function" approach, similar to the Minimum Phone Error (MPE) training used in state-of-the-art speech recognition systems.

Experimental Setup

The system was initially trained on 24.6 hours of clean Mandarin speech. It was then tested on a challenging archive of 33 hours of noisy, spontaneous lectures from National Taiwan University. This setup created a significant acoustic mismatch, with the initial word accuracy being only approximately 50.26%. The evaluation used 80 manually selected Chinese single-word queries.

Results and Discussion

The baseline system, without any feedback, achieved a Mean Average Precision (MAP) of 0.4819. The experimental results demonstrate that even a minimal amount of user feedback can lead to significant improvements.

Performance comparison of different objective functions using feedback from the top 5 results (N=5).

As shown in the results for the short-term scenario, providing feedback on just the top 5 results using the $F_4$ objective function improved the MAP to 0.4979. When the feedback set was expanded to the top 20 results, the MAP reached 0.5040. In the long-term scenario, learning from 60 previous queries improved the MAP for new, unseen queries to 0.5073.

The $F_4$ objective function consistently outperformed the others, as it balances ranking improvements with a smoothing term that prevents the model from diverging too far based on limited data. Furthermore, the authors noted that the small size of the feedback data makes on-line retraining computationally feasible for real-time applications.

Conclusion

This research highlights that optimizing for relative ranking rather than absolute transcription accuracy is a powerful strategy for Spoken Term Detection in mismatched environments. By leveraging a small amount of user feedback, the proposed discriminative training framework significantly enhances retrieval performance. Future work will explore the use of noisy implicit feedback (such as click-through data) and extending the system to handle more complex multi-word queries.

Summary updated

BibTeX
@inproceedings{lee2010improved,
  author = {Hung-yi Lee and Chia-Ping Chen and Ching-feng Yeh and Lin-Shan Lee},
  title = {{Improved spoken term detection by discriminative training of acoustic models based on user relevance feedback}},
  booktitle = {Interspeech},
  year = {2010},
  url = {https://openreview.net/forum?id=BIyziIwu4Mz},
}
Figure