Neural Skill Transfer from Supervised Language Tasks to Reading Comprehension

Todor Mihaylov, Zornitsa Kozareva, Anette FrankarXiv 2017 · arXiv

Abstract

Reading Comprehension (RC) is a complex linguistic task that requires the simultaneous application of multiple lower-level skills, such as entity recognition and logical inference. Traditional models typically attempt to learn these skills implicitly from massive datasets, which is both computationally expensive and data-intensive. This paper introduces the Skillful Reader, a modular architecture that explicitly transfers knowledge from specialized supervised tasks—Named Entity Recognition (NER), Question Type Classification (QTC), Textual Entailment (TE), and Paraphrase Detection—to the RC task. By utilizing pre-trained encoders and a novel token-wise supervision method, the Skillful Reader demonstrates significant improvements in training efficiency and performance, particularly in data-scarce scenarios.

Introduction

State-of-the-art Reading Comprehension models often rely on large-scale datasets like SQuAD to learn the nuances of language. However, the requirement for over 100,000 labeled examples poses a significant barrier in domains where data is scarce. Furthermore, training these models from scratch is inefficient, as they must "re-invent" basic linguistic capabilities. We address these challenges by leveraging existing supervised datasets for simpler tasks to "jump-start" a complex RC model, treating linguistic skills as modular components.

Methodology

Skill Learning and Token-Wise Supervision

The core of our approach involves pre-training Bi-directional LSTM (Bi-LSTM) encoders on four distinct skill tasks: NER, QTC, TE, and Paraphrase Detection. A critical innovation is token-wise deep label supervision. For classification tasks like QTC, the model is trained to predict the label at every token rather than just once for the entire sentence. This forces the encoder to learn the specific contribution of individual words to the overall category, making the resulting representations more granular and useful for the downstream RC task.

Neural Skill Transfer

The Skillful Reader architecture integrates these pre-trained encoders into a primary RC model. As shown in Figure 1, the representations from the skill encoders are passed through "adapter" layers and concatenated with the main task's internal representations. This modular design allows the model to leverage specialized knowledge without retraining the base encoders from scratch.

Figure 1: Architecture of the Skillful Reader, illustrating the integration of skill-specific encoders via adapter layers into the main reading comprehension pipeline.

Experimental Setup

We evaluated the Skillful Reader using the SQuAD dataset as the primary RC benchmark. Skill encoders were trained on CoNLL 2012 (NER), TREC (QTC), SNLI (TE), and PPDB 2.0 (Paraphrase). We specifically investigated the model's performance under conditions of extreme data scarcity, using subsets as small as 2% of the SQuAD training data.

Results and Discussion

Performance and Data Efficiency

The inclusion of all skill modules improved the SQuAD F1-score from a baseline of 59.41 to 61.67 after fine-tuning. More importantly, the Skillful Reader exhibited remarkable efficiency in the "cold start" phase of training. When restricted to small subsets of the training data, the model reached its peak performance significantly faster than the baseline, as illustrated in Figure 2.

Figure 2: F-score over training steps using only 5% of the training data, demonstrating the significant efficiency gains provided by pre-trained skill tasks.

Impact of Individual Skills

Analysis of individual skills revealed that Textual Entailment and Paraphrase Detection provided the most significant boosts. This aligns with the nature of Reading Comprehension, which often involves identifying a passage that either paraphrases the question or provides an entailing context for the answer.

Conclusion

The Skillful Reader demonstrates that modularity and targeted skill transfer can significantly mitigate the data requirements of complex NLP tasks. By supervising models at the token level during skill acquisition, we ensure that transferred knowledge is highly relevant to the granular demands of reading comprehension. Future work will explore the application of this framework to more advanced architectures and the inclusion of additional linguistic skills.

Summary updated

Abstract

Reading comprehension is a challenging task in natural language processing and requires a set of skills to be solved. While current approaches focus on solving the task as a whole, in this paper, we propose to use a neural network `skill' transfer approach. We transfer knowledge from several lower-level language tasks (skills) including textual entailment, named entity recognition, paraphrase detection and question type classification into the reading comprehension model. We conduct an empirical evaluation and show that transferring language skill knowledge leads to significant improvements for the task with much fewer steps compared to the baseline model. We also show that the skill transfer approach is effective even with small amounts of training data. Another finding of this work is that using token-wise deep label supervision for text classification improves the performance of transfer learning.
BibTeX
@misc{mihaylov2017neural,
  author = {Todor Mihaylov and Zornitsa Kozareva and Anette Frank},
  title = {{Neural Skill Transfer from Supervised Language Tasks to Reading Comprehension}},
  year = {2017},
  eprint = {1711.03754},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1711.03754},
}
Figure