Abstract
Machine Reading Comprehension (MRC) models have achieved high accuracy on standard benchmarks but remain vulnerable to adversarial perturbations. This paper presents a multi-task learning framework designed to enhance MRC robustness in "noisy" documents. By combining hierarchical knowledge enrichment—utilizing external sources like WordNet—with a dual-task objective that includes unanswerability detection, the proposed model achieves superior performance on adversarial datasets such as AddSent and AddOneSent.
1. Introduction
While state-of-the-art MRC models perform well on clean datasets, they are often "brittle," relying on lexical overlaps rather than semantic understanding. This makes them susceptible to "distractor" sentences—adversarial additions that mislead models without confusing humans. Furthermore, many models are biased toward always providing an answer, even when a question is unanswerable based on the text. We address these issues by creating a model that is both more knowledgeable through external data and more skeptical through explicit unanswerability detection.
2. Methodology
The proposed architecture follows an encode-align-decode framework, enhanced by two key innovations:
2.1 Hierarchical Knowledge Enhancement Layer
To move beyond the provided text, the model integrates external background knowledge at three distinct levels:
- Low Level: Semantic connections between query and passage words are identified using WordNet.
- Medium Level: Auxiliary "supportive knowledge" is selected based on the specific query task.
- High Level: A fusion strategy integrates this external knowledge into the contextual representations of the document and query.
2.2 Multi-Task Learning
The model is trained simultaneously on two sub-tasks:
- Answer Span Prediction: Identifying the start and end indices of the answer using a pointer network.
- Unanswerability Prediction: A "sentinel identifier" calculates the probability that a question cannot be answered. If this probability exceeds a threshold ($\gamma$), the question is flagged as unanswerable.
The base architecture utilizes GloVe and ELMo embeddings, character-level embeddings, and bi-LSTMs for contextual encoding.
3. Experimental Evaluation
3.1 Datasets
The model was evaluated on standard and adversarial benchmarks:
- SQuAD 1.1 & 2.0: Used for standard performance evaluation and training unanswerability examples.
- AddSent: An adversarial dataset featuring human-like but irrelevant distractor sentences.
- AddOneSent: An adversarial dataset where a random distracting sentence is appended to the passage.
3.2 Results
The experimental results, summarized in Table 1, highlight the model's robustness.
The proposed model significantly outperforms previous baselines on adversarial tasks. On AddSent, it achieved an F1 score of 65.5, compared to the previous best of 60.1 (KAR). On AddOneSent, it reached 72.5 F1. While the model shows a slight performance trade-off on the "clean" SQuAD 1.1 Dev set (73.9 EM / 82.4 F1) compared to specialized models like SLQA, its superior generalizability on noisy data confirms the effectiveness of the multi-task approach.
4. Conclusion and Future Work
This work demonstrates that combining hierarchical knowledge with joint training for unanswerability significantly improves MRC robustness against adversarial distractions. Future research will focus on closing the performance gap on clean datasets while maintaining this robustness and exploring the sensitivity of the unanswerability threshold.
Summary updated