DialDoc 2022 Shared Task: Open-Book Document-grounded Dialogue Modeling

Song Feng, Siva Patel, Hui WanACL 2022

TL;DR

This paper introduces the DialDoc 2022 Shared Task, focusing on building open-book, goal-oriented dialogue systems that can ground their responses in multiple documents. The task challenges models to generate natural language agent utterances based on dialogue history and a set of domain documents, even when the grounding document isn't explicitly known. The results show significant improvements over baselines, with top teams achieving substantial gains in evaluation metrics.

What Problem Does This Paper Address?

Existing goal-oriented dialogue systems often struggle with real-world scenarios where conversations can draw information from multiple, potentially unknown, documents. Previous work, like the first DialDoc workshop, focused on single-document grounding. This paper addresses the gap by proposing an "open-book, closed-domain" setting where dialogues can be grounded in any of the provided domain documents, mimicking more complex, information-seeking interactions. The challenge lies in developing systems that can effectively retrieve and synthesize information from multiple sources to generate relevant and coherent agent responses.

Methods

The DialDoc 2022 Shared Task focuses on generating agent responses in natural language, given dialogue history, the current user turn, and a set of domain documents. The core problem is tackled by systems that typically employ a pipeline of:

  • Retrieval: Identifying relevant documents or passages from the provided corpus. This often involves techniques like Dense Passage Retrieval (DPR) or Approximate Nearest Neighbor (ANN) methods.
  • Re-ranking: Further refining the retrieved passages to select the most pertinent ones. This can involve using cross-encoder models (e.g., BERT, RoBERTa) for more accurate scoring.
  • Generation: Synthesizing the information from the selected passages into a natural language agent utterance. Models like BART (leveraging a sequence-to-sequence architecture) are commonly used, often enhanced with grounding span predictors.

Some approaches also incorporate multi-task learning, data augmentation, and curriculum learning to improve performance.

Datasets

The shared task utilizes the MultiDoc2Dial dataset, introduced in Feng et al. (2021). This dataset contains conversations grounded in multiple documents across four domains. For the DialDoc 2022 task, two evaluation settings were created:

  • SEEN: Test data from domains present in the training data.
  • UNSEEN: Test data from a domain not seen during training (cdccovid domain).

The dataset includes dialogue turns with speaker roles, dialogue acts, grounding text spans, and human-generated utterances.

Key Insights and Ideas

  • Multi-document grounding is crucial: The success of top-performing systems highlights the importance of handling conversations that draw from multiple documents, a more realistic scenario than single-document grounding.
  • Pipeline architectures are effective: A combination of retriever, re-ranker, and generator modules consistently yields strong results, suggesting that breaking down the problem into distinct stages is beneficial.
  • Pre-training and fine-tuning advancements: Leveraging powerful pre-trained models like BERT, RoBERTa, and BART, and adapting them through techniques like FiD (Fusion-in-Decoder) and Child-Tuning, significantly boosts performance.
  • The UNSEEN setting is a significant challenge: The performance drop on the UNSEEN task compared to the SEEN task underscores the difficulty of generalizing to entirely new domains, even with a closed-domain constraint.

Key Results

The DialDoc 2022 Shared Task saw significant improvements over the baseline.

  • SEEN Task: The best-performing system (CPII-NLP) achieved 52.06 F1 and a total score of 191.30, compared to the baseline's 35.85 F1 and 126.21 total score.
  • UNSEEN Task: The top system (CPII-NLP) achieved 34.65 F1 and a total score of 130.79, significantly outperforming the baseline's 19.26 F1 and 59.52 total score.

These results demonstrate that participants were able to effectively build systems that ground dialogue in multiple documents.

Limitations and Future Work

  • Intrinsic vs. Extrinsic Evaluation: While automatic metrics (F1, SacreBLEU, METEOR, RougeL) provide a quantitative ranking, human evaluation was also conducted to assess relevance and fluency, indicating that automatic metrics alone may not fully capture system quality.
  • Generalization to Unseen Domains: The performance gap between SEEN and UNSEEN tasks suggests that further research is needed to improve model generalization to entirely new domains.
  • Data Augmentation and Regularization: Techniques like data augmentation and regularization were explored to improve reader accuracy and robustness, indicating ongoing efforts to refine model training.
  • Future work likely involves exploring more sophisticated retrieval and generation strategies, better handling of unseen domains, and potentially end-to-end trainable models.

Summary updated

BibTeX
@inproceedings{feng2022dialdoc,
  author = {Song Feng and Siva Patel and Hui Wan},
  title = {{DialDoc 2022 Shared Task: Open-Book Document-grounded Dialogue Modeling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure