TL;DR
This paper introduces a dialogue strategy for speech-based information retrieval systems that tackles vague user queries. By dynamically asking clarifying questions based on information gain, the system effectively narrows down search results from large, unstructured knowledge bases, significantly improving retrieval success.
What Problem Does This Paper Address?
Spoken dialogue interfaces offer convenience, but users often speak before fully formulating their queries, leading to vague or incomplete inputs. This is exacerbated by Automatic Speech Recognition (ASR) errors. In large document retrieval systems, these imprecise queries can result in an overwhelming number of matched documents, making it difficult for users to find what they need. Existing approaches often rely on structured knowledge bases with predefined slots, which are not suitable for general document collections like newspaper articles or support manuals. This paper addresses the challenge of clarifying and constraining user queries in speech-driven retrieval from large, unstructured knowledge bases.
Methods
The core of the proposed solution is a clarification dialogue strategy that dynamically selects the most informative question to ask the user. The system uses Information Gain (IG) as a criterion to measure how efficiently a question can reduce the set of potential documents.
The strategy involves:
- Initial Retrieval: The system first retrieves documents based on the user's initial query.
- Question Generation: A pool of candidate questions is prepared using three methods:
- Dependency Structure Analysis (Method 1): This bottom-up approach analyzes document texts to identify words that are likely to cause ambiguity (e.g., verbs with many possible objects). It extracts modifier-head pairs and calculates entropy to determine which words are most uncertain, leading to questions like "What did you delete?"
- Metadata (Method 2): This top-down approach leverages metadata associated with documents (e.g., product names). Questions are generated to clarify which product or version the user is interested in, such as "What is the version of your Windows?"
- Human Knowledge (Method 3): This method uses hand-crafted questions based on common knowledge from support call centers, aiming to capture key information like "When did the symptom occur?"
- Optimal Question Selection: The system calculates the IG for each candidate question and selects the one with the highest IG (above a certain threshold).
- Query Update: The user's reply to the selected question is used to update and refine the original query sentence. This updated query is then used for subsequent retrieval.
- Iteration: The process repeats, with the system asking new questions based on the updated query until a satisfactory result is achieved or a threshold is met.
Datasets
The paper uses a software support knowledge base (KB) provided by Microsoft Corporation. This KB consists of:
- Glossary
- Frequently Asked Questions (FAQ)
- Support articles
The total size of the KB is approximately 40,000 documents. The paper does not introduce a new benchmark dataset but evaluates its system on this existing KB.
Key Insights and Ideas
- Information Gain for Dialogue: Using Information Gain as a metric to select clarifying questions is a powerful way to prioritize questions that most effectively reduce ambiguity and narrow down search results.
- Hybrid Question Generation: Combining bottom-up (dependency analysis) and top-down (metadata, human knowledge) approaches for generating candidate questions creates a robust and comprehensive question pool. This allows the system to address different types of ambiguity.
- Query Refinement is Crucial: Simply accepting ASR results for vague queries is insufficient. Dynamically updating the query sentence based on user replies is key to improving retrieval accuracy.
- Effectiveness of Specific Questions: While general questions (like those from human knowledge) are used more often, specific questions derived from dependency structure analysis are more effective per question, suggesting a trade-off between question frequency and impact.
Key Results
The experimental evaluation demonstrated significant improvements in retrieval success rate:
- Simulation: The proposed method achieved a 12.6% improvement in retrieval success rate compared to using only the ASR result.
- Log Data (Real Dialogue): The proposed method achieved a 7.7% improvement in retrieval success rate compared to using only the ASR result.
- The average rank of the correct document also improved substantially, from 7.45 (ASR baseline) to 4.40 (proposed method, log data) and 3.85 (proposed method, simulation).
- Using all three question generation methods together yielded the best results, with a success rate of 83.3%.
Limitations and Future Work
- Domain Specificity: The authors acknowledge that the proposed approach is intended for restricted domains where knowledge sources are available and not directly applicable to open-domain information retrieval like general web search.
- ASR Errors in Replies: The system updates the query sentence based on user replies regardless of ASR errors in those replies. While confirming ASR errors is possible, it was omitted to avoid complicating the dialogue.
- User Misunderstanding: Users sometimes misunderstood system questions, leading to uninformative replies. The authors suggest making questions more specific or providing reply examples to address this.
- Future Work: The authors suggest exploring more specific questions and potentially displaying reply examples to improve user interaction and further enhance retrieval efficiency. They also believe the methodology is adaptable to other restricted domain information retrieval tasks.
Summary updated