A Multi-Task Learning Machine Reading Comprehension Model for Noisy Document (Student Abstract)

Zhijing Wu, Hua Xu

Abstract

Machine Reading Comprehension (MRC) models have achieved high accuracy on standard benchmarks but remain vulnerable to adversarial perturbations. This paper presents a multi-task learning framework designed to enhance MRC robustness in "noisy" documents. By combining hierarchical knowledge enrichment—utilizing external sources like WordNet—with a dual-task objective that includes unanswerability detection, the proposed model achieves superior performance on adversarial datasets such as AddSent and AddOneSent.

1. Introduction

While state-of-the-art MRC models perform well on clean datasets, they are often "brittle," relying on lexical overlaps rather than semantic understanding. This makes them susceptible to "distractor" sentences—adversarial additions that mislead models without confusing humans. Furthermore, many models are biased toward always providing an answer, even when a question is unanswerable based on the text. We address these issues by creating a model that is both more knowledgeable through external data and more skeptical through explicit unanswerability detection.

2. Methodology

The proposed architecture follows an encode-align-decode framework, enhanced by two key innovations:

2.1 Hierarchical Knowledge Enhancement Layer

To move beyond the provided text, the model integrates external background knowledge at three distinct levels:

  • Low Level: Semantic connections between query and passage words are identified using WordNet.
  • Medium Level: Auxiliary "supportive knowledge" is selected based on the specific query task.
  • High Level: A fusion strategy integrates this external knowledge into the contextual representations of the document and query.

2.2 Multi-Task Learning

The model is trained simultaneously on two sub-tasks:

  1. Answer Span Prediction: Identifying the start and end indices of the answer using a pointer network.
  2. Unanswerability Prediction: A "sentinel identifier" calculates the probability that a question cannot be answered. If this probability exceeds a threshold ($\gamma$), the question is flagged as unanswerable.

The base architecture utilizes GloVe and ELMo embeddings, character-level embeddings, and bi-LSTMs for contextual encoding.

3. Experimental Evaluation

3.1 Datasets

The model was evaluated on standard and adversarial benchmarks:

  • SQuAD 1.1 & 2.0: Used for standard performance evaluation and training unanswerability examples.
  • AddSent: An adversarial dataset featuring human-like but irrelevant distractor sentences.
  • AddOneSent: An adversarial dataset where a random distracting sentence is appended to the passage.

3.2 Results

The experimental results, summarized in Table 1, highlight the model's robustness.

Table 1: Performance comparison of the proposed model against state-of-the-art baselines on standard and adversarial datasets.

The proposed model significantly outperforms previous baselines on adversarial tasks. On AddSent, it achieved an F1 score of 65.5, compared to the previous best of 60.1 (KAR). On AddOneSent, it reached 72.5 F1. While the model shows a slight performance trade-off on the "clean" SQuAD 1.1 Dev set (73.9 EM / 82.4 F1) compared to specialized models like SLQA, its superior generalizability on noisy data confirms the effectiveness of the multi-task approach.

4. Conclusion and Future Work

This work demonstrates that combining hierarchical knowledge with joint training for unanswerability significantly improves MRC robustness against adversarial distractions. Future research will focus on closing the performance gap on clean datasets while maintaining this robustness and exploring the sensitivity of the unanswerability threshold.

Summary updated

Abstract

Current neural models for Machine Reading Comprehension (MRC) have achieved successful performance in recent years. However, the model is too fragile and lack robustness to tackle the imperceptible adversarial perturbations to the input. In this work, we propose a multi-task learning MRC model with a hierarchical knowledge enrichment to further improve the robustness for noisy document. Our model follows a typical encode-align-decode framework. Additionally, we apply a hierarchical method of adding background knowledge into the model from coarse-to-fine to enhance the language representations. Besides, we optimize our model by jointly training the answer span and unanswerability prediction, aiming to improve the robustness to noise. Experiment results on benchmark datasets confirm the superiority of our method, and our method can achieve competitive performance compared with other strong baselines.
BibTeX
@inproceedings{wu2020multi,
  author = {Zhijing Wu and Hua Xu},
  title = {{A Multi-Task Learning Machine Reading Comprehension Model for Noisy Document (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i10.7254},
}
Figure