TL;DR
Dialogue systems often struggle with unanswerable queries, resorting to generic "I don't know" responses that frustrate users. This paper introduces a novel approach to generate contextually aware and engaging "don't know" responses, improving user interaction by making the system seem more intelligent and less monotonous.
What Problem Does This Paper Address?
Modern dialogue systems, even sophisticated ones, inevitably encounter queries outside their knowledge domain. When this happens, they typically fall back on canned, unhelpful responses like "I don't know." This not only fails to address the user's underlying intent but also leads to a poor user experience, reducing engagement and making the system feel unintelligent. The challenge is to create a fallback mechanism that acknowledges the query's context, politely declines to answer, and maintains a natural, engaging conversation flow, rather than simply stating ignorance.
Methods
The paper proposes two main approaches to generate these contextualized fallback responses:
- Dependency Based Approach (DBA): This method leverages linguistic parsing. It uses hand-crafted dependency templates that analyze the grammatical structure of a user's query. By mapping these structures to pre-defined "don't know" response templates, the system can generate responses that are more specific to the query's topic. This approach also incorporates rules for handling pronouns, named entities, and auxiliary verbs to ensure grammatical correctness and contextual relevance. The DBA serves as a baseline and also helps in generating training data.
- Sequence-to-Sequence Approach: Recognizing the scalability limitations of the rule-based DBA, the authors employ a pre-trained T5 transformer model. To avoid overfitting on the DBA's handcrafted responses, they create a dataset by pairing paraphrased questions (from the Quora Question Pairs dataset) with the "don't know" responses generated by the DBA. This "I Don't Know Dataset" (IDKD) is then used to fine-tune the T5 model. This approach aims to leverage the transformer's inherent world knowledge and its ability to generate more natural and paraphrastic responses.
Datasets
- Quora Question Pairs (QQP) dataset: Used to generate paraphrases of input questions. Specifically, question pairs labeled as "1" or deemed similar were utilized.
- "I Dont Know Dataset" (IDKD): A newly created dataset of approximately 100,000 (query, fallback response) pairs. This dataset was generated by pairing paraphrased questions from QQP with the "don't know" responses produced by the Dependency Based Approach.
Key Insights and Ideas
- "Saying No" as an Art: The core idea is that a polite and contextually aware refusal is more effective than a generic "I don't know." This reframes the fallback mechanism from a failure state to an opportunity for better user interaction.
- Leveraging Linguistic Structure: The DBA demonstrates that even rule-based systems, when grounded in linguistic parsing, can generate surprisingly relevant and contextualized fallback responses.
- Synthetic Data for Deep Learning: The paper highlights the effectiveness of using synthetically generated data (from the DBA) to train powerful sequence-to-sequence models like T5, especially when real-world data for specific tasks is scarce.
- Paraphrasing for Robustness: By pairing paraphrased questions with fallback responses, the T5 model learns to generate more varied and robust "don't know" answers that are less tied to the exact phrasing of the original query.
- Pre-trained Models as Knowledge Bases: The T5 model's ability to generate more natural-sounding responses is attributed to its pre-training, which imbues it with a broad understanding of language and world knowledge.
Key Results
- Grammaticality: The Sequence-to-Sequence (T5) approach achieved a higher percentage of grammatically correct responses (87.2%) compared to the Dependency Based Approach (81.6%) in human evaluations.
- Relevance: While the T5 model's relevance score (3.66) was slightly lower than the DBA's (3.97), it was still considered good. The DBA was noted for being highly relevant but limited in coverage.
- Coverage: The DBA could only handle 54.5% of random QQP queries, whereas the Seq2Seq approach achieved 100% coverage.
- Novelty: The T5 model generated, on average, double the number of novel words compared to the DBA, indicating more varied and less templated responses.
- Qualitative Examples: Table 4 showcases examples where the T5 model's responses are more nuanced and contextually aware, often incorporating elements not explicitly mentioned by the user, thanks to its pre-training.
Limitations and Future Work
- Relevance vs. Paraphrasing Trade-off: The authors acknowledge that as the model gains more paraphrasing power, there's a slight drop in the direct relevance of the "don't know" response. This suggests a delicate balance between being informative (even in refusal) and being concise.
- Scalability of DBA: The Dependency Based Approach, while precise, has limited coverage and is not easily scalable to a vast array of query types.
- Future Work: The paper suggests that these approaches can serve as effective baselines. Future work could explore further enhancing the paraphrasing ability of the fallback system while maintaining conciseness, and integrating these contextualized fallback mechanisms into larger dialogue design frameworks.
Summary updated