Introduction
Searching through spoken content, such as podcasts or broadcast news, presents unique challenges compared to traditional text-based retrieval. First, Automatic Speech Recognition (ASR) is inherently imperfect, introducing noise into the searchable index. Second, users cannot easily "skim" audio results; listening to multiple audio files to find relevant information is time-consuming and cognitively demanding.
While interactive information retrieval (IIR) can help clarify user intent, existing systems often rely on fixed rules or limited interaction types. This paper proposes a framework that uses a Markov Decision Process (MDP) to dynamically choose between different types of interactions—each with varying costs to the user—to maximize retrieval accuracy while minimizing user effort.
The MDP Framework for Interactive Retrieval
The authors model the search session as a sequential decision-making problem. The system must decide at each turn which action will most effectively improve the search results relative to the "cost" of the user's time and effort.
Action Set and Optimization
The system can choose from four distinct actions:
- Return Documents: The system presents a list; the user selects a relevant document, and the system performs relevance feedback.
- Return Keyterm: The system asks a low-effort "Yes/No" question about a specific keyword's relevance.
- Request: The system asks for a new, supplementary query—a high-effort action that provides significant information.
- Show List: The terminal action where final results are presented, ending the session.
The policy for choosing these actions is trained via Reinforcement Learning. The reward function balances the improvement in retrieval quality (measured by Mean Average Precision, or MAP) against the negative rewards (costs) associated with user effort for each interaction type.
State Estimation
A critical component of this system is the State Estimation module. Since the system does not have ground-truth knowledge of result quality during a live session, it must "guess" the current performance level. The authors utilize a multi-class SVM classifier based on features like query length, clarity scores, and the statistical distribution of retrieval scores to estimate the current state.
Experimental Setup
The approach was evaluated using a Mandarin broadcast news archive consisting of 5,047 stories (approx. 96 hours). To test robustness against ASR noise, two transcription conditions were used: Doc (I) with 45.64% character accuracy and Doc (II) with 52.15% accuracy. The evaluation involved 163 query sets provided by 22 graduate students.
Key Results and Analysis
The MDP-based approach demonstrated significant improvements over baseline strategies that utilized fixed interaction patterns.
As shown in the results table, the MDP approach (both "Oracle" and "Estimate") consistently outperformed baselines in both MAP and "Final Return" (a metric accounting for user cost). Under "Oracle" conditions—where the system has perfect knowledge of its performance—the MAP reached 0.4607 for Doc I, compared to a 0.3703 baseline of no interaction. Even with imperfect state estimation, the system maintained a clear advantage, achieving a MAP of 0.4335.
Insights
- Economic Trade-off: The core innovation is treating interaction as an economic trade-off. The system aims for the highest accuracy per unit of user effort.
- Context-Dependent Strategy: The optimal action shifts based on the current state. High-ambiguity results may justify the high cost of a "Request" action early on, while better initial results might only require a low-cost "Keyterm" check.
- Robustness to Noise: The performance gains were consistent across both low and high ASR accuracy conditions, indicating that the MDP policy effectively manages speech recognition errors.
Conclusion and Future Work
This research highlights that a principled, automated approach to choosing interaction types can significantly enhance spoken content retrieval. However, the gap between "Oracle" and "Estimated" performance suggests that improving real-time state estimation is a primary bottleneck. Future work may involve moving beyond simulated users to real-world testing and exploring more complex, nuanced interaction types.
Summary updated