Introduction
Current progress in Natural Language Processing (NLP) is often characterized by specialized models designed for narrow tasks. While state-of-the-art performance is achieved in silos like translation or sentiment analysis, these models lack the versatility of a general-purpose agent. Salesforce Research introduces the Natural Language Decathlon (decaNLP) to address this, framing ten diverse tasks as a single problem: question answering.
The decaNLP Framework
The central premise of decaNLP is that any NLP task can be mapped to a (context, question, answer) triplet. By using natural language questions to define the task, a model can leverage its linguistic understanding to determine the required output format and logic.
The benchmark includes ten datasets covering a wide spectrum of linguistic challenges, ranging from semantic parsing (WikiSQL) and machine translation (IWSLT) to sentiment analysis (SST) and pronoun resolution (MWSC). This variety forces models to develop a more robust and general internal representation of language.
The Multitask Question Answering Network (MQAN)
To tackle decaNLP, the authors propose the Multitask Question Answering Network (MQAN). Unlike traditional multitask models that use task-specific output heads, MQAN employs a unified multi-pointer-generator decoder. This architecture allows the model to generate words from a fixed vocabulary or copy them directly from either the context or the question.
This flexibility is critical for generalization. For instance, in summarization, the model primarily copies from the context. In sentiment analysis, the model might copy the labels "positive" or "negative" directly from the question prompt itself.
Key Insights and Model Behavior
A significant finding is the model's reliance on the "question pointer." Analysis of the decoder's behavior reveals that for tasks like sentiment analysis (SST) or natural language inference (MNLI), the model learns to point almost exclusively to the question to retrieve the correct label.
Furthermore, the authors discovered that training follows an "anti-curriculum" pattern. Performance improved when the model was first trained on the most foundational task—reading comprehension via SQuAD—before engaging in joint multitask training. This suggests that "learning to read" is a prerequisite for broader linguistic generalization across other tasks like translation or parsing.
Experimental Results
MQAN demonstrates strong performance across the decaNLP suite:
- State-of-the-Art: It achieved SOTA results on WikiSQL in a single-task setting, despite not being specialized for semantic parsing.
- Zero-Shot Generalization: The model showed an ability to perform sentiment analysis on unseen domains (Yelp, Amazon) with high accuracy (~80%) without specific training on those datasets.
- Transfer Learning: Pretraining on decaNLP significantly accelerated learning and improved final performance on new tasks, such as Named Entity Recognition, compared to starting from random initialization.
Conclusion
The Natural Language Decathlon and the MQAN architecture represent a significant shift toward more general NLP systems. By unifying tasks through a question-answering interface, the model can share knowledge across domains and generalize to new tasks through natural language instructions. While a "multitask gap" remains between unified and specialized models, decaNLP provides a robust benchmark for the future of general-purpose language understanding.
Summary updated