TL;DR
This paper tackles the challenge of building multilingual document-grounded dialogue systems, where responses need to be generated in various languages based on provided documents. The authors propose a three-stage approach (retrieval, re-ranking, generation) enhanced with techniques like negative sample augmentation, prompt learning, and adversarial training, achieving a strong second-place finish in the DialDoc23 Shared Task.
What Problem Does This Paper Address?
Transferring document-grounded dialogue (DGD) models, which are designed to answer questions and converse based on provided documents, from high-resource languages (like English) to low-resource languages is a significant hurdle. This challenge is amplified when the system needs to handle multilingual documents and generate responses in multiple languages. Existing DGD datasets often focus on plain text, overlooking the importance of structural document information. This work aims to bridge this gap by developing a robust multilingual DGD system capable of understanding and responding to queries across different languages, grounded in provided documents.
Methods
The authors adopt a three-stage pipeline:
- Retrieval: The initial step involves retrieving relevant passages from a document based on the dialogue history and current query. They employ a contrastive learning approach, enhancing it with negative sample augmentation to improve retrieval performance.
- Re-ranking: To further refine the retrieved passages, a re-ranking model is used. They fine-tune large multilingual models like XLM-ROBERTa and InfoXLM. Adversarial training (using FGM) is applied to boost generalization. An ensemble of these models is used to predict passage scores.
- Generation: The final stage generates the response. They fine-tune the MT5 model, a massively multilingual pre-trained text-to-text transformer. Key techniques here include:
- Multilingual Prompts: Simple yet effective prompts are added to the input to guide the model in generating responses in the correct target language (e.g., "La réponse française qui en résulte est <response>" for French).
- Adversarial Training: Techniques like FGM and AWP are used to improve the model's robustness and generalization.
- Pseudo-labeling: The fine-tuned model is used to generate labels for the test set, which are then used to further fine-tune the model.
Datasets
The primary dataset used is the DialDoc23 Shared Task dataset. This dataset comprises:
- Vietnamese dialogues (797 dialogues, 3,446 turns)
- French dialogues (816 dialogues, 3,510 turns)
- A corpus of 17,272 paragraphs, with each dialogue turn grounded in a paragraph from this corpus.
The authors also mention using FrDoc2BotRerank and ViDoc2BotRerank datasets for re-ranking and FrDoc2BotGeneration and ViDoc2BotGeneration datasets for generation, which appear to be subsets or specific versions of the main DialDoc dataset tailored for these tasks.
Key Insights and Ideas
- Negative Sample Augmentation is Crucial for Retrieval: Expanding the set of negative examples during training significantly improves the performance of the retrieval model.
- Multilingual Prompts Guide Generation: Explicitly adding language-specific prefixes to the input during generation effectively guides multilingual models like MT5 to produce responses in the desired language, mitigating cross-lingual interference.
- Ensembling Re-rankers Boosts Performance: While an ensemble of re-ranking models might show a slight drop on the dev-test set, it can lead to substantial improvements on the final test set.
- Adversarial Training Enhances Robustness: Applying adversarial training techniques (FGM, AWP) consistently improves model performance across re-ranking and generation stages.
Key Results
The proposed method achieved a second-place ranking in the DialDoc23 Shared Task, with a total score of 210.25.
Specific results on the dev-test set include:
- Retrieval: The "pre-batch-neg" method improved F1 from 58.39 to 59.54.
- Re-ranking: XLM-ROBERTa achieved F1 of 63.14, Bleu of 49.23, and Rouge of 60.78. The ensemble of XLM-ROBERTa and InfoXLM showed a slight drop on dev-test but a 2.05-point improvement on the final test set.
- Generation: The final model (GS/Adv/prompt/PL) achieved an F1 of 70.14, Bleu of 60.98, and Rouge of 68.26 on the dev-test set.
Limitations and Future Work
The authors acknowledge that due to time and equipment constraints, they could not explore all potential improvements. They suggest several promising future directions:
- Cross-lingual Data Translation: Translating official Chinese and English data into Vietnamese and French, and then using all available data for pre-training.
- Weighted Ranking: Combining retrieved top-k passages with re-ordered relevancy for weighted ranking.
- Combined Training Tasks: Exploring training strategies that combine reordering tasks with build tasks.
- Dichotomous Models: Investigating models that score generated statements to select the best response.
Summary updated