Docalog: Multi-document Dialogue System using Transformer-based Span Retrieval

Sayed Hesam Alavian, Ali Satvaty, Sadra Sabouri, Ehsaneddin Asgari, Hossein SametiACL 2022

TL;DR

This paper introduces Docalog, a three-stage system for multi-document dialogue question answering that excels at identifying relevant knowledge within a collection of documents. Docalog achieved strong results on the DialDoc-22 challenge by effectively combining document retrieval, answer span prediction, and an ultimate span selection mechanism.

What Problem Does This Paper Address?

Creating AI systems that can engage in natural, informative conversations with humans has been a long-standing goal in artificial intelligence. A key challenge in this area is building dialogue systems that can answer user questions by drawing information from multiple documents. This is particularly difficult because the system must first identify which documents are relevant to the user's query and then pinpoint the exact answer within those documents. The DialDoc Shared Task specifically focuses on this multi-document dialogue question answering problem, requiring systems to understand conversation flow, identify relevant knowledge, and generate coherent answers.

Methods

Docalog employs a three-stage pipeline:

  1. Document Retriever (DR. TEIT): This component identifies the most relevant documents from a larger corpus based on the user's current question and dialogue history. It uses a pre-trained language-agnostic BERT sentence embedding (LaBSE) and incorporates both document titles and TF-IDF-based cosine similarity for retrieval. A key innovation here is the inclusion of dialogue history and the emphasis on title tokens, which are found to be crucial when the conversation context shifts.
  2. Answer Span Prediction Model: This stage takes the top-k retrieved documents and uses a fine-tuned RoBERTa language model to predict potential answer spans within each document. To handle dialogue history, the last two turns of the conversation are appended to the current question. Notably, this model is pre-trained on several conversational question answering (CQA) datasets (CoQA, QuAC, DoQA, Doc2Dial) before being fine-tuned on the MultiDoc2Dial dataset. This pre-training helps it learn general QA capabilities and mitigates the impact of retrieval errors during training.
  3. Ultimate Span Picker: Since the answer span probabilities from different documents aren't directly comparable, this final component ranks the predicted spans to select the most likely answer. It uses a convex combination of two similarity measures: embedding-based similarity and character-level TF-IDF similarity between the query and the predicted span. This approach aims to leverage both semantic and lexical matching.

Datasets

The primary dataset used is MultiDoc2Dial, which is derived from the Doc2Dial dataset. It consists of documents and conversations grounded in those documents. The system was trained and evaluated on this dataset, specifically on the MDD-SEEN and MDD-UNSEEN folds for both development and testing. The span prediction model also underwent pre-training on several CQA datasets, including:

  • CoQA (Reddy et al., 2019)
  • QuAC (Choi et al., 2018)
  • DoQA (Campos et al., 2020)
  • Doc2Dial (Feng et al., 2020b)

Key Insights and Ideas

  • Dialogue History is Crucial: Incorporating dialogue history into both the document retrieval and span prediction stages significantly improves performance, especially in multi-document settings where context can shift.
  • Title Importance: Document titles are surprisingly effective signals for identifying relevant documents, particularly when the conversation's focus changes.
  • Pre-training for Robustness: Pre-training the span prediction model on a variety of CQA datasets before fine-tuning on the target task helps to decouple retrieval errors from span prediction learning, leading to a more robust system.
  • Hybrid Similarity for Span Selection: Combining embedding-based and character-level TF-IDF similarities in the ultimate span picker provides a more comprehensive way to rank potential answers.

Key Results

In the test phase of the MultiDoc2Dial 2022 challenge, Docalog achieved:

  • MDD-SEEN: F1-score of 36.07% and SacreBLEU score of 23.70%.
  • MDD-UNSEEN: F1-score of 28.44% and SacreBLEU score of 20.52%.

The paper notes that Docalog@1 (using the top document retrieved) achieved the best F1 score, suggesting that the ultimate span picker could still benefit from further improvements. The document retriever (DR. TEIT) demonstrated strong performance with a Precision@5 of 86% and an MRR of 0.72, indicating that relevant documents are consistently found within the top retrieved results.

Limitations and Future Work

The authors acknowledge that the ultimate span picker could be further improved. While Docalog@1 performed best, indicating the retriever is effective, the final selection mechanism might not be fully optimized. Future work could focus on enhancing this component. The paper also mentions the availability of their implementation on GitHub, encouraging further research and development in this area.

Summary updated

BibTeX
@inproceedings{alavian2022docalog,
  author = {Sayed Hesam Alavian and Ali Satvaty and Sadra Sabouri and Ehsaneddin Asgari and Hossein Sameti},
  title = {{Docalog: Multi-document Dialogue System using Transformer-based Span Retrieval}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure