Abstract
Typically, machine learning systems solve new tasks by training on thousands of examples. In contrast, humans can solve new tasks by reading some instructions, with perhaps an example or two. To take a step toward closing this gap, we introduce a framework for developing NLP systems that solve new tasks after reading their descriptions, synthesizing prior work in this area. We instantiate this framework with a new English language dataset, ZEST, structured for task-oriented evaluation on unseen tasks. Formulating task descriptions as questions, we ensure each is general enough to apply to many possible inputs, thus comprehensively evaluating a model's ability to solve each task. Moreover, the dataset's structure tests specific types of systematic generalization. We find that the state-of-the-art T5 model achieves a score of 12% on ZEST, leaving a significant challenge for NLP researchers.
Introduction
The current paradigm in Natural Language Processing (NLP) relies heavily on "learning from examples," where models are trained on thousands of labeled pairs for specific tasks like sentiment analysis or summarization. While effective, this approach suffers from high data requirements, a tendency for models to exploit statistical shortcuts, and a lack of flexibility compared to human learning. Humans can often perform a new task after reading a simple set of instructions. This paper introduces ZEST (ZEro Shot learning from Task descriptions), a benchmark designed to evaluate whether NLP models can achieve similar zero-shot reasoning capabilities by learning directly from natural language task descriptions.
The ZEST Framework
The authors propose a formal framework where a model produces an output $y$ given an input $x$ and a task description $d_\tau$. Unlike standard supervised learning where the task is implicit in the training set, here the task itself is a variable provided at inference time.
To measure performance rigorously, the authors introduce Task Competence (C@T). Rather than averaging accuracy across individual instances, C@T calculates the percentage of entire tasks where the model's performance exceeds a threshold $T$ (e.g., 90% F1). This metric ensures that a model has truly mastered the underlying logic of a task across various inputs.
The ZEST Dataset
ZEST consists of 25,026 triples (task description, input, answer) covering 1,251 unique tasks across domains such as U.S. Presidents, Dog Breeds, and National Parks. Each task is paired with 20 different passages to ensure consistency. The dataset is specifically structured to test four types of systematic generalization: Paraphrase, Semantic Flips, Composition, and Output Structure.
Experimental Results
The researchers evaluated state-of-the-art models, including T5-11B and BART-large, using ZEST and Multi-Task Learning (MTL) on existing QA datasets.
The results highlight a significant "instruction gap." While the T5-11B + MTL model achieved a mean F1 of 56%, its Task Competence at the 90% threshold (C@90) was only 12%. In comparison, a human baseline reached 42% C@90. Performance was particularly poor on tasks requiring "Semantic Flips"—where a single word change in the instruction changes the required output—and "Output Structure," where models struggled to generate JSON-formatted responses for unseen task types.
Discussion and Future Work
The low performance of current state-of-the-art models on ZEST suggests that being "good at QA" does not equate to being "good at following instructions." Models often fail to truly understand the instruction, instead relying on keyword recognition. Future research should explore few-shot learning (providing a few examples alongside the description), meta-learning architectures for instruction mapping, and improved multi-step reasoning for complex compositional tasks.
Summary updated