Task-Oriented Document-Grounded Dialog Systems by HLTPR@RWTH for DSTC9 and DSTC10

David Thulke, Nico Daheim, Christian Dugast, Hermann Ney

This paper summarizes the award-winning contributions of the HLTPR group at RWTH Aachen University to the 9th and 10th Dialog System Technology Challenges (DSTC). The authors tackle the complexities of document-grounded dialog, where a system must move beyond structured databases to answer user queries using unstructured text like FAQs or reviews.

TL;DR

The authors present a robust framework for document-grounded dialog that significantly improves efficiency—achieving up to a 100x speedup in document selection—while enhancing response factuality through noisy channel modeling and retrieval-augmented generation. Their work highlights the critical need for better evaluation metrics, as standard scores like BLEU show poor correlation with human judgment in spoken dialog contexts.

What Problem Does This Paper Address?

Traditional task-oriented dialog systems are often "brittle," limited to information stored in rigid, structured databases. When a user asks a question outside that schema (e.g., "What is the policy on service animals?"), the system fails. Document-grounding allows systems to "read" unstructured text to find answers.

However, three major hurdles remain:

  1. Efficiency: Comparing a user's query against thousands of documents in real-time is computationally expensive.
  2. Robustness: Systems trained on clean, written text often fall apart when faced with noisy Automatic Speech Recognition (ASR) transcripts from spoken conversations.
  3. Factuality: Generative models (like BART) are prone to "hallucinations," creating responses that sound fluent but are factually incorrect.

Methods

The authors break the task into three subtasks: Detection (is this a knowledge-seeking turn?), Selection (which document has the answer?), and Generation (what is the response?).

  • Hierarchical & Bi-Encoder Selection: To solve the efficiency problem, they moved away from slow "cross-encoders" (which compare the query to every document one-by-one). Instead, they proposed Hierarchical Selection (narrowing down by Domain $\rightarrow$ Entity $\rightarrow$ Document) and Bi-Encoders, which pre-compute document embeddings for lightning-fast retrieval.
  • Retrieval Augmented Generation (RAG): Rather than forcing the model to look at only one document, they use RAG to condition the response on the top $n$ retrieved documents, reducing the impact of selection errors.
  • Noisy Channel Model: To improve factuality, they used Bayes’ Theorem to reformulate the generation task. Instead of just predicting the next word, the model also considers how well a generated response "explains" the source document.
  • Style Adaptation: For spoken dialog (DSTC10), they introduced special style tokens (<written> vs. <spoken>) to help the model transition from formal training data to the more casual, disfluent nature of speech.

Datasets

  • MultiWOZ 2.1: The foundational dataset for task-oriented dialog.
  • DSTC9/10 Data: Extensions of MultiWOZ including new domains (e.g., "attraction" in San Francisco) and a mix of written and spoken (ASR-generated) transcripts.
  • Data Augmentation: The authors generated 16,675 new samples by automatically pairing FAQ documents with existing dialog contexts to improve model robustness.

Key Insights and Ideas

  • Efficiency is not a Zero-Sum Game: Hierarchical selection provided a 24x speedup while actually improving accuracy by reducing the search space and potential for confusion among similar documents.
  • The Metric Gap: One of the most striking findings is that for spoken dialog (DSTC10), standard metrics like BLEU and ROUGE had zero or even negative correlation with human judgments. They found that factuality-specific metrics (like $Q^2$) are far more reliable.
  • ASR Robustness: Models are highly sensitive to formatting. Simply "cleaning" training data to match ASR output (removing punctuation, lowercasing, and writing out numbers like "42" as "forty-two") significantly boosted performance on spoken tasks.

Key Results

  • Speed: Their Bi-Encoder approach achieved a 100x speedup over the hierarchical model and a 2,500x speedup over the baseline cross-encoder.
  • DSTC9: Their refined system outperformed all challenge finalists in the selection and generation tasks.
  • DSTC10: The team placed 1st in the generation subtask based on automatic metrics and 3rd in human evaluation.
  • Detection: Data augmentation and in-domain pre-training improved the Detection F1 score from 75.3 to 84.1 on noisy data.

Limitations and Future Work

While Bi-Encoders are much faster, they still slightly underperform compared to Cross-Encoders in terms of raw accuracy. The authors suggest that further improvements in contrastive learning (training the model to better distinguish between "near-miss" documents) could close this gap. Additionally, as the field moves toward DSTC11, the authors note the need to handle subjective knowledge (like user reviews) and scenarios where multiple documents are required to answer a single query.

Summary updated

Abstract

This paper summarizes our contributions to the document-grounded dialog tasks at the 9th and 10th Dialog System Technology Challenges (DSTC9 and DSTC10). In both iterations the task consists of three subtasks: first detect whether the current turn is knowledge seeking, second select a relevant knowledge document, and third generate a response grounded on the selected document. For DSTC9 we proposed different approaches to make the selection task more efficient. The best method, Hierarchical Selection, actually improves the results compared to the original baseline and gives a speedup of 24x. In the DSTC10 iteration of the task, the challenge was to adapt systems trained on written dialogs to perform well on noisy automatic speech recognition transcripts. Therefore, we proposed data augmentation techniques to increase the robustness of the models as well as methods to adapt the style of generated responses to fit well into the proceeding dialog. Additionally, we proposed a noisy channel model that allows for increasing the factuality of the generated responses. In addition to summarizing our previous contributions, in this work, we also report on a few small improvements and reconsider the automatic evaluation metrics for the generation task which have shown a low correlation to human judgments.
BibTeX
@article{thulke2024task,
  author = {David Thulke and Nico Daheim and Christian Dugast and Hermann Ney},
  title = {{Task-Oriented Document-Grounded Dialog Systems by HLTPR@RWTH for DSTC9 and DSTC10}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2024},
  url = {https://openreview.net/forum?id=neTRAPHKBQ},
}

Loading related content...

Figure