TL;DR
This paper introduces Task2Dial, a new dataset and task for commonsense-enhanced, document-grounded task-based dialogues. It addresses the challenge of creating dialogue systems that can not only follow instructions from documents but also handle clarification questions requiring real-world common sense, moving beyond simple question-answering.
What Problem Does This Paper Address?
Traditional goal-oriented dialogue systems often rely on structured databases, which are limiting when domain knowledge is primarily in unstructured formats like documents. While document-grounded dialogue systems have emerged, they typically focus on question answering or strictly following document-based instructions. This leaves a gap in handling the natural, unpredictable nature of human conversations where users might ask clarifying questions that aren't explicitly in the document but require common sense to answer. For instance, in a cooking recipe, a user might ask about a safe way to handle a hot pan, a detail not always present in the recipe itself. Task2Dial aims to bridge this gap by creating dialogues that mimic these real-world scenarios, where an "Information Giver" (IG) uses a document to instruct an "Information Follower" (IF), and the IF can ask commonsense-based questions.
Methods
The core of this work is the introduction of the Task2Dial task and dataset. The task is framed as an instruction-following scenario where an IG, with access to a document (in this case, recipes), guides an IF through a task. The IG can simplify, rephrase, or omit information from the original document. Crucially, the IF can ask clarification questions that may require commonsense knowledge beyond the document's content. The IG must then provide answers that are both grounded in the document (when applicable) and informed by common sense.
The dataset was created through a multi-stage pilot study and then by employing research assistants as annotators. This approach aimed to ensure higher quality and more detailed dialogues compared to large-scale crowdsourcing. Annotators were given detailed instructions and examples to create dialogues that were not only task-oriented but also captured natural variability and commonsense reasoning.
Datasets
The paper introduces the Task2Dial dataset, which consists of:
- A set of recipe documents.
- Conversations between an Information Giver (IG) and an Information Follower (IF) grounded in these recipes.
The dataset was compiled from highly-rated recipes with positive feedback from public sources. The authors compare Task2Dial to the Doc2Dial dataset, noting that Task2Dial has a significantly higher average number of turns per dialogue (18.15 vs. 12.94) and longer utterances.
Key Insights and Ideas
- Bridging the Gap Between Document-Grounded and Commonsense Dialogue: Task2Dial explicitly aims to combine instruction following from documents with the need for commonsense reasoning, a crucial aspect of natural human interaction that is often overlooked in existing datasets.
- Real-World Unpredictability in Dialogue: The task is designed to emulate the "off-script" nature of human conversations, where users might deviate from a strict instructional path to ask for clarification or advice based on their understanding of the world.
- The Importance of Flexible Dialogue Management: The ability to handle questions not directly grounded in the document necessitates more flexible dialogue state tracking and planning mechanisms.
- Lexical Richness and Variation: The Task2Dial dataset exhibits higher lexical richness and variation compared to Doc2Dial, posing a challenge for training models but promising more natural and less template-like system utterances.
Key Results
The paper focuses on introducing the task and dataset, rather than presenting extensive quantitative results from a trained model. However, they provide descriptive statistics:
- Dialogue Size: Task2Dial dialogues average 18.15 turns, with 19.79 tokens per turn.
- Lexical Richness: Task2Dial has a Type-token ratio (TTR) of 0.025 and a Mean segmental TTR (MSTTR) of 0.84.
- Infrequent Words: A significant portion of bigrams (59%) and trigrams (75%) in Task2Dial appear only once, highlighting the challenge of out-of-vocabulary words and the need for robust language models.
Limitations and Future Work
- Limitations: The paper acknowledges that the dialogues are text-based, and future work could explore spoken dialogue. The dataset is currently focused on cooking tasks, and expansion to other domains would be beneficial. The high proportion of infrequent words presents a training challenge.
- Future Work: The authors plan to benchmark and evaluate dialogue systems on the Task2Dial dataset, potentially extending their existing "Chefbot" system. They also suggest extending this approach to human-robot interaction (HRI) scenarios. Further research into dialogue management that can handle commonsense reasoning and flexible instruction-giving is encouraged.
Summary updated