Sure, here's a blog-style summary of the paper "Consistency Training by Synthetic Question Generation for Conversational Question Answering":
TL;DR
This paper introduces CoTaH, a novel method for conversational question answering (CQA) that tackles the challenge of irrelevant historical context. By generating synthetic questions to augment the conversation history and employing consistency training, CoTaH makes CQA models more robust to noise, leading to improved performance, especially in longer conversations.
What Problem Does This Paper Address?
Conversational question answering (CQA) systems aim to answer user questions within the context of an ongoing dialogue. Unlike traditional QA, CQA relies heavily on the conversation history to understand ambiguous questions. However, this history can also contain irrelevant information that can confuse the model and degrade its performance. Previous approaches have attempted to filter this irrelevant history, but often rely on "gold answers" (ground truth) from the history, which isn't realistic in real-world scenarios where models must use their own predictions. This paper addresses the gap of making CQA models robust to irrelevant history without relying on gold answers, by leveraging synthetic data and consistency training.
Methods
The core of CoTaH is a two-stage pipeline:
- History Augmentation Module:
- Synthetic Question Generation (CQG): A generative transformer model (based on BART-Large) is trained to generate synthetic questions. It takes the document, previous history, and the current answer as input to generate a new question.
- Question Filtering & Injection (QS): A crucial step to prevent noise. The generated synthetic questions are filtered based on their similarity to the existing history and the current question using LaBSE embeddings. Only a subset of relevant synthetic questions are selected and combined with the original history to form an augmented history (H). This filtering is key to ensuring the synthetic data is helpful and not detrimental.
- Question Answering Module:
- Consistency Training: The CQA model (a BERT-based architecture in this case) is trained to produce consistent answers regardless of whether it uses the original history (Hk) or the augmented history (H). This consistency is enforced using KL-divergence loss between the answer distributions generated from both histories.
- Model-Agnostic: The approach is designed to be model-agnostic, meaning it can be applied to various CQA architectures.
Datasets
The paper uses the QuAC (Question Answering in Context) dataset for training and evaluation. The authors note that they did not test on CoQA (another CQA dataset) to facilitate direct comparison with prior work that exclusively uses QuAC. They also describe a custom data splitting strategy for QuAC's development set to create a dev/test split, ensuring no data leakage between dialogs.
Key Insights and Ideas
- Synthetic data as a noise reducer: Generating synthetic questions that are relevant to the conversation flow, but not necessarily identical to existing history, can help the model learn to distinguish between useful and noisy historical context.
- Consistency is key: Forcing the model to produce the same answer from both the original and augmented history implicitly trains it to ignore irrelevant parts of the history.
- Filtering is crucial: Simply augmenting history with any synthetic questions isn't enough. The filtering mechanism to remove highly similar or irrelevant synthetic questions is vital for performance gains.
- Later turns benefit most: The improvements are most pronounced in later turns of a conversation, where historical context is richer and more prone to containing irrelevant information.
Key Results
- Overall Improvement: CoTaH-BERT outperforms the BERT baseline by 1.8% in F1 score.
- Unanswerable Questions: CoTaH-BERT shows a considerable improvement in detecting unanswerable questions, indicating better language understanding.
- Performance on Long Histories: The improvement is primarily attributed to better performance on questions with substantial historical context, as visualized in Figure 2 (Appendix D) and discussed in the text.
- Question Generation Quality: The synthetic question generation module achieves good scores on standard metrics like BLEU-1, BLEU-4, and Rouge-L, indicating the quality of generated questions.
- Ablation Studies: Filtering synthetic questions and using a threshold for applying augmentation (τ) significantly improve performance. A uniform distribution for selecting synthetic questions was found to be slightly better than a linear one.
Limitations and Future Work
- Question Generation Overhead: The initial phase of generating synthetic questions adds computational cost. While this is done offline, it's a necessary preprocessing step.
- Dataset Specificity: The research is limited to the QuAC dataset. Future work should explore performance on other CQA datasets like CoQA.
- Large Language Models: The study did not extensively test on state-of-the-art large language models like ChatGPT. Further experiments are needed to understand CoTaH's placement relative to these models.
- Real-world Deployment: While the method is model-agnostic, its effectiveness with the most advanced LLMs needs further investigation.
Summary updated