The Natural Language Decathlon: Multitask Learning as Question Answering

Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, Richard SocherarXiv 2018 · arXiv

Introduction

Current progress in Natural Language Processing (NLP) is often characterized by specialized models designed for narrow tasks. While state-of-the-art performance is achieved in silos like translation or sentiment analysis, these models lack the versatility of a general-purpose agent. Salesforce Research introduces the Natural Language Decathlon (decaNLP) to address this, framing ten diverse tasks as a single problem: question answering.

The decaNLP Framework

The central premise of decaNLP is that any NLP task can be mapped to a (context, question, answer) triplet. By using natural language questions to define the task, a model can leverage its linguistic understanding to determine the required output format and logic.

Examples of NLP tasks framed as question answering.

The benchmark includes ten datasets covering a wide spectrum of linguistic challenges, ranging from semantic parsing (WikiSQL) and machine translation (IWSLT) to sentiment analysis (SST) and pronoun resolution (MWSC). This variety forces models to develop a more robust and general internal representation of language.

The Multitask Question Answering Network (MQAN)

To tackle decaNLP, the authors propose the Multitask Question Answering Network (MQAN). Unlike traditional multitask models that use task-specific output heads, MQAN employs a unified multi-pointer-generator decoder. This architecture allows the model to generate words from a fixed vocabulary or copy them directly from either the context or the question.

This flexibility is critical for generalization. For instance, in summarization, the model primarily copies from the context. In sentiment analysis, the model might copy the labels "positive" or "negative" directly from the question prompt itself.

Key Insights and Model Behavior

A significant finding is the model's reliance on the "question pointer." Analysis of the decoder's behavior reveals that for tasks like sentiment analysis (SST) or natural language inference (MNLI), the model learns to point almost exclusively to the question to retrieve the correct label.

Analysis of pointer usage across different decaNLP tasks.

Furthermore, the authors discovered that training follows an "anti-curriculum" pattern. Performance improved when the model was first trained on the most foundational task—reading comprehension via SQuAD—before engaging in joint multitask training. This suggests that "learning to read" is a prerequisite for broader linguistic generalization across other tasks like translation or parsing.

Experimental Results

MQAN demonstrates strong performance across the decaNLP suite:

  • State-of-the-Art: It achieved SOTA results on WikiSQL in a single-task setting, despite not being specialized for semantic parsing.
  • Zero-Shot Generalization: The model showed an ability to perform sentiment analysis on unseen domains (Yelp, Amazon) with high accuracy (~80%) without specific training on those datasets.
  • Transfer Learning: Pretraining on decaNLP significantly accelerated learning and improved final performance on new tasks, such as Named Entity Recognition, compared to starting from random initialization.

Conclusion

The Natural Language Decathlon and the MQAN architecture represent a significant shift toward more general NLP systems. By unifying tasks through a question-answering interface, the model can share knowledge across domains and generalize to new tasks through natural language instructions. While a "multitask gap" remains between unified and specialized models, decaNLP provides a robust benchmark for the future of general-purpose language understanding.

Summary updated

Abstract

Deep learning has improved performance on many natural language processing (NLP) tasks individually. However, general NLP models cannot emerge within a paradigm that focuses on the particularities of a single metric, dataset, and task. We introduce the Natural Language Decathlon (decaNLP), a challenge that spans ten tasks: question answering, machine translation, summarization, natural language inference, sentiment analysis, semantic role labeling, zero-shot relation extraction, goal-oriented dialogue, semantic parsing, and commonsense pronoun resolution. We cast all tasks as question answering over a context. Furthermore, we present a new Multitask Question Answering Network (MQAN) jointly learns all tasks in decaNLP without any task-specific modules or parameters in the multitask setting. MQAN shows improvements in transfer learning for machine translation and named entity recognition, domain adaptation for sentiment analysis and natural language inference, and zero-shot capabilities for text classification. We demonstrate that the MQAN's multi-pointer-generator decoder is key to this success and performance further improves with an anti-curriculum training strategy. Though designed for decaNLP, MQAN also achieves state of the art results on the WikiSQL semantic parsing task in the single-task setting. We also release code for procuring and processing data, training and evaluating models, and reproducing all experiments for decaNLP.
BibTeX
@misc{mccann2018natural,
  author = {Bryan McCann and Nitish Shirish Keskar and Caiming Xiong and Richard Socher},
  title = {{The Natural Language Decathlon: Multitask Learning as Question Answering}},
  year = {2018},
  eprint = {1806.08730},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1806.08730},
}
Figure