TL;DR
This paper tackles the challenge of building goal-oriented dialogue systems that can effectively leverage external documents to generate relevant responses. They propose novel input representations for pre-trained language models (RoBERTa and BART) that incorporate dialogue acts, document titles, and answer span rankings, significantly improving performance on the Dialdoc21 shared task. Additionally, they introduce a curriculum learning strategy to progressively train their models on increasingly difficult examples.
What Problem Does This Paper Address?
Creating dialogue systems that can engage in goal-oriented conversations, like booking a flight or answering a complex question, often requires access to external information. The challenge lies in how to effectively integrate this external knowledge, typically in the form of documents, into the dialogue generation process. Specifically, the Dialdoc21 shared task focuses on two subtasks: first, identifying relevant text spans within a document that correspond to a user's utterance, and second, generating a coherent and informative response based on these identified spans and the dialogue history. Existing approaches often struggle to fully exploit the rich structural and semantic information present in documents and dialogue history, leading to less accurate knowledge grounding and less natural responses.
Methods
The authors propose two main models, one for each subtask, built upon pre-trained language models:
- Subtask 1 (Knowledge Span Prediction): They use RoBERTa-large as a backbone and enhance its input representation by:
- Dialogue Act Prediction: Incorporating dialogue act labels to guide the model in identifying relevant spans.
- Title Embeddings: Adding embeddings that represent the title of the document section from which a span originates. This helps group related spans.
- MRC Architecture: Employing a standard Machine Reading Comprehension (MRC) architecture with a Bi-GRU layer for predicting span start and end positions.
- Curriculum Learning: Implementing a curriculum learning strategy to train the model on progressively harder examples.
- Subtask 2 (Response Generation): They leverage BART-base and enrich its input with:
- Special Tokens: Introducing custom tokens like
<user>,<agent>,<doc>,<title>,<rank>,<u>, and<h>to demarcate different parts of the input (dialogue history, document, titles, etc.). - Type-of-Input Embeddings: Distinguishing between dialogue history and document content.
- Rank Embeddings: Representing the ranking of title spans predicted by the subtask 1 model.
- Rank-in-Section Embeddings: Representing the ranking of answer spans within each title.
- Curriculum Learning: Applying the same curriculum learning strategy as in subtask 1.
The curriculum learning approach involves dividing the training data into difficulty levels based on teacher models' performance and then training the main model in stages, starting with easier examples and gradually introducing more complex ones.
Datasets
The primary dataset used is doc2dial, which is designed for goal-oriented dialogue grounded in documents. It comprises conversations associated with documents and is divided into four domains: DMV, VA, SSA, and studentaid. The dataset is split into train, validation, and test-dev sets. For evaluation, a test dataset encompassing these four domains plus an unseen domain (COVID-19) was used. The test-dev dataset represents 30% of the test dataset, excluding the unseen domain.
Key Insights and Ideas
- Leveraging Document Structure is Crucial: Simply feeding raw document text is insufficient. Explicitly incorporating structural information like document titles and the hierarchical relationships between spans (rank within a title, rank of the title itself) significantly boosts performance.
- Dialogue Acts as a Strong Signal: The dialogue act of an utterance provides valuable context for identifying relevant knowledge spans, acting as a strong supervisory signal.
- Curriculum Learning for Robustness: Training models with a curriculum, starting from simpler examples and progressing to harder ones, leads to more robust and better-performing models, especially in complex tasks like document-grounded dialogue.
- Pre-trained Models are Powerful Backbones: RoBERTa and BART, when augmented with these novel input representations, demonstrate substantial improvements, highlighting the effectiveness of transfer learning in this domain.
Key Results
- Subtask 1: The proposed span prediction model achieved an F1-score of 74.81 on the test-dev phase (ranked 7th) and 73.41 on the test phase (ranked 5th). The addition of dialogue acts, title embeddings, and curriculum learning progressively improved performance.
- Subtask 2: The response generation model achieved sacreBLEU scores of 37.50 on the test-dev phase (ranked 3rd) and 41.06 on the test phase (ranked 1st). Each added embedding type (Type-of-Input, Rank, Rank-in-Section) and curriculum learning contributed to significant gains in sacreBLEU.
Limitations and Future Work
- Limitations: The paper doesn't explicitly detail limitations, but the reliance on pre-trained models and specific dataset structures suggests potential challenges in generalization to vastly different domains or document types. The effectiveness of curriculum learning is also dependent on the quality of the teacher models and the defined difficulty levels.
- Future Work: The authors suggest that their proposed input representations and curriculum learning strategy can be applied to other document-grounded dialogue tasks. Further exploration into more sophisticated ways of representing document structure and dialogue context could lead to even greater improvements.
Summary updated