Abstract
This paper summarizes our contributions to the document-grounded dialog tasks at the 9th and 10th Dialog System Technology Challenges (DSTC9 and DSTC10). In both iterations the task consists of three subtasks: first detect whether the current turn is knowledge seeking, second select a relevant knowledge document, and third generate a response grounded on the selected document. For DSTC9 we proposed different approaches to make the selection task more efficient. The best method, Hierarchical Selection, actually improves the results compared to the original baseline and gives a speedup of 24x. In the DSTC10 iteration of the task, the challenge was to adapt systems trained on written dialogs to perform well on noisy automatic speech recognition transcripts. Therefore, we proposed data augmentation techniques to increase the robustness of the models as well as methods to adapt the style of generated responses to fit well into the proceeding dialog. Additionally, we proposed a noisy channel model that allows for increasing the factuality of the generated responses. In addition to summarizing our previous contributions, in this work, we also report on a few small improvements and reconsider the automatic evaluation metrics for the generation task which have shown a low correlation to human judgments.
Introduction
Traditional task-oriented dialog systems are often limited by rigid, structured databases. When a user query falls outside the predefined schema—such as asking about a specific policy or a subjective review—these systems typically fail. Document-grounded dialog systems address this by "reading" unstructured text (FAQs, manuals, or reviews) to provide answers. The HLTPR group at RWTH Aachen University developed a robust framework for these challenges during the DSTC9 and DSTC10 competitions, focusing on three primary hurdles: computational efficiency, robustness to noisy speech input, and the factuality of generated responses.
Methodology: The Three-Subtask Framework
The system architecture is divided into three distinct subtasks: Detection, Selection, and Generation.
1. Knowledge-Seeking Turn Detection
The first step determines if a user's turn requires external knowledge. To improve performance on noisy data, the team utilized data augmentation, generating 16,675 new samples by pairing FAQ documents with existing dialog contexts. Pre-training on in-domain data and specific text-cleaning techniques (lowercasing, removing punctuation) boosted the Detection F1 score from 75.3 to 84.1 on spoken transcripts.
2. Efficient Knowledge Selection
In DSTC9, the baseline "cross-encoder" approach—comparing a query against every document individually—was computationally prohibitive. The team proposed Hierarchical Selection, which narrows the search space through a Domain $\rightarrow$ Entity $\rightarrow$ Document hierarchy. This method provided a 24x speedup while simultaneously improving accuracy by reducing potential confusion between similar documents. Further optimization using Bi-Encoders, which pre-compute document embeddings, achieved up to a 2,500x speedup over the baseline.
3. Grounded Response Generation
To mitigate model "hallucinations," the team employed Retrieval Augmented Generation (RAG), conditioning responses on the top $n$ retrieved documents. For DSTC10, which involved noisy Automatic Speech Recognition (ASR) transcripts, they introduced style adaptation tokens (<written> vs. <spoken>) to help the model transition from formal training data to casual, disfluent speech. A Noisy Channel Model based on Bayes’ Theorem was also implemented to ensure generated responses were more factually grounded in the source documents.
Evaluation and the Metric Gap
A significant finding of this work is the discrepancy between automatic metrics and human judgment. In the context of spoken dialog (DSTC10), standard metrics like BLEU and ROUGE showed near-zero or even negative correlation with human assessments of appropriateness and accuracy. This highlights the necessity for factuality-specific metrics, such as $Q^2$, which proved more reliable.
Key Results and Conclusion
The HLTPR systems demonstrated state-of-the-art performance across both challenges:
- DSTC9: The refined system outperformed all finalists in selection and generation.
- DSTC10: The team placed 1st in the generation subtask according to automatic metrics and 3rd in human evaluation.
- Efficiency: The Bi-Encoder approach achieved a 100x speedup over the hierarchical model without significant loss in accuracy.
Future work will focus on closing the accuracy gap between Bi-Encoders and Cross-Encoders using contrastive learning and expanding the system's capability to handle subjective knowledge and multi-document reasoning.
Summary updated