Doc2Dial: A Framework for Dialogue Composition Grounded in Documents

Song Feng, Kshitij P. Fadnis, Q. Vera Liao, Luis A. LastrasAAAI 2020 · DOI

Abstract

Building effective task-oriented dialogue agents requires massive amounts of chat logs, which are often scarce or proprietary. Doc2Dial is an end-to-end framework designed to bridge the gap between static business documents (like manuals and policies) and the training data needed for goal-oriented chatbots. By automatically extracting "preconditions" and "solutions" from text and simulating dialogue flows, it enables the efficient collection of high-quality, document-grounded conversation data through an innovative asynchronous crowdsourcing process.

Introduction

Companies possess vast repositories of manuals, FAQs, and policy documents that contain the information agents need. However, these documents are structured for reading, not for the back-and-forth diagnostic process of a conversation. Doc2Dial addresses this "data bottleneck" by creating a pipeline that transforms raw documents into structured dialogue flows, which are then used to collect natural human utterances.

The Doc2Dial Pipeline

The authors propose a modular pipeline that moves from raw text to a full conversation through automated and human-assisted steps.

The Doc2Dial framework architecture, illustrating the flow from document processing to automated labeling (Auto) and human annotation (Anno).

Automated Labeling and Composition

  1. TEXTAUTO: The system parses documents (often HTML) to understand their structure. It identifies "discourse connectives" (like "if" or "as long as") to automatically label text spans as either preconditions (the context/criteria) or solutions (the answer/action).
  2. RELAUTO: This module identifies how these spans relate to one another using heuristics based on document proximity and tree structures.
  3. DIALAUTO: This "engine" treats a conversation as a sequence of states. It randomly selects content from the pool of identified preconditions and solutions, assigns a role (Agent or User), and assigns a "dialogue act" (e.g., request, respond, or disagree).

Human-in-the-Loop Refinement

While the "Auto" modules provide a baseline, the framework includes integrated tools (TEXTANNO, RELANNO, DIALANNO) for human annotators to refine labels and flesh out simulated flows into natural language.

The Doc2Dial crowdsourcing interface, showing the asynchronous process where workers review document context and dialogue history to provide natural language responses.

A standout feature is the asynchronous crowdsourcing model. Unlike traditional methods that require two people to be online simultaneously to "chat," Doc2Dial allows workers to create and evaluate individual turns independently. This removes the "timing constraint" of dialogue collection, making the process much easier to scale.

Key Insights and Results

The framework frames goal-oriented dialogue as a "diagnostic process." The agent's job is to identify which preconditions in a document apply to the user's current state to reach the correct solution. By leveraging inherent document structures like HTML headers, the framework can infer logical flow without needing hand-crafted decision trees.

In evaluation, the framework successfully generated 5 dialogue flows per document across a 2,000-document set. In a pilot study, 67% of the turns were labeled as a "match" by crowd workers. Qualitative analysis showed that when crowd workers understood the context of the document, they were highly effective at producing natural, relevant utterances.

Limitations and Future Work

The primary limitation identified was "mismatches" in the dialogue flows, mostly traced back to errors in the initial automated labeling of preconditions and solutions. Future work involves improving the accuracy of the TEXTAUTO and RELAUTO modules through more advanced NLP techniques. Since its introduction, the Doc2Dial dataset has become a major benchmark for the research community.

Summary updated

BibTeX
@inproceedings{feng2020doc,
  author = {Song Feng and Kshitij P. Fadnis and Q. Vera Liao and Luis A. Lastras},
  title = {{Doc2Dial: A Framework for Dialogue Composition Grounded in Documents}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i09.7089},
}
Figure