TL;DR
This paper introduces QAInfomax, a novel regularizer for question answering (QA) systems that maximizes mutual information between passages, questions, and answers. By encouraging deeper understanding beyond superficial correlations, QAInfomax significantly improves robustness against adversarial distractors and achieves state-of-the-art performance on the challenging Adversarial-SQuAD benchmark.
What Problem Does This Paper Address?
Modern reading comprehension systems, while achieving high accuracy on standard datasets, often exhibit a superficial understanding of language. They tend to exploit spurious correlations within the data, making them vulnerable to "distractor sentences" that are semantically related but do not actually contain the answer. This fragility is highlighted by datasets like Adversarial-SQuAD, where existing state-of-the-art models degrade significantly. The core problem is that current QA models don't truly "understand" the question and passage in a way that distinguishes genuine answers from plausible-sounding but incorrect ones.
Methods
The paper proposes QAInfomax, a regularizer that enhances existing QA models by maximizing the mutual information (MI) among the question, passage, and answer. This is achieved by incorporating principles from Deep Infomax (DIM), a technique for learning representations by maximizing MI.
QAInfomax introduces two main constraints:
- Local Constraint (LC): This constraint ensures that each word representation within the answer span contains information about the other answer words and their surrounding context. It aims to prevent models from focusing on isolated keywords and encourages a holistic understanding of the answer.
- Global Constraint (GC): This constraint pushes the summarized answer representation to share information globally with all other question and passage representations. This encourages the model to learn representations that are robust and not overly reliant on local, superficial cues.
The MI estimation is performed using a Jensen-Shannon divergence-based approach, which is efficiently implemented with cross-entropy loss. The overall objective function combines the standard answer span prediction loss with the QAInfomax regularization term.
Datasets
- SQuAD (Stanford Question Answering Dataset): Used for pre-training the BERT model.
- Adversarial-SQuAD: The primary benchmark dataset for evaluating the robustness of the proposed QAInfomax model. This dataset is an adversarial version of SQuAD, created by adding distractor sentences to paragraphs, specifically designed to challenge QA systems' understanding.
Key Insights and Ideas
- Beyond Superficial Correlations: The paper strongly argues that high accuracy on standard QA datasets doesn't necessarily equate to deep understanding. Models can learn "shortcuts" by exploiting superficial word co-occurrences, which QAInfomax aims to mitigate.
- Mutual Information as a Regularizer: The core idea of using MI maximization as a regularizer for QA is novel. It provides a principled way to encourage models to learn more robust and semantically grounded representations.
- Local vs. Global Context: The distinction between the Local Constraint (focusing on answer span and its immediate context) and the Global Constraint (considering the entire question and passage) is crucial. Both are shown to be important for achieving the best results.
- Robustness to Adversarial Examples: The significant improvement on ADDSENT and ADDONESENT metrics demonstrates that QAInfomax makes models more resilient to carefully crafted adversarial examples, a critical aspect for real-world deployment.
Key Results
- State-of-the-Art Performance: QAInfomax, when applied to a BERT-base model, achieved state-of-the-art performance on the Adversarial-SQuAD dataset, outperforming strong baselines.
- Significant Improvement on Adversarial Metrics:
- BERT-base achieved an F1 score of 51.0 on ADDSENT and 63.4 on ADDONESENT.
- BERT-base + QAInfomax achieved an F1 score of 54.5 on ADDSENT and 64.9 on ADDONESENT, with the improvement on ADDSENT being statistically significant (p-value < 0.05).
- No Hurt to Original Performance: The proposed QAInfomax regularizer improved robustness without degrading the model's performance on the original SQuAD dataset.
- Ablation Study: An ablation study confirmed that both the Local and Global Constraints are essential for the model's effectiveness.
- Training Speed: The GC objective slightly degraded training speed by approximately 28%, indicating a trade-off between robustness and computational efficiency.
Limitations and Future Work
- Training Speed: The Global Constraint, which involves averaging MI over many representations, increases computational cost and slows down training.
- Future Work: The authors suggest investigating more efficient methods to reduce the limitations of QAInfomax and further improve the generalization capability of QA systems. They also mention that exploring different parameterizations of MI is left as future work.
Summary updated