Abstract
Modern NLP relies heavily on large-scale supervised learning from labeled examples. However, humans can perform new tasks simply by reading a single sentence of instruction. In "Learning from Task Descriptions," Weller et al. (2020) propose a shift toward this human-like capability. They introduce ZEST (ZEro Shot learning from Task descriptions), a benchmark designed to evaluate how well models generalize to unseen tasks based solely on natural language descriptions. Their findings reveal a significant "competence gap": while humans achieve 42% success on complex tasks, even the largest models reach only 12%.
Introduction: The Limitations of Example-Based Learning
The dominant paradigm in Natural Language Processing (NLP) is supervised learning, where models are trained on thousands of input-output pairs. While effective, this approach suffers from three critical issues:
- Scalability: It is impossible to curate massive datasets for every potential task a user might conceive.
- Accessibility: Domain experts should be able to interact with AI using natural language rather than requiring data labeling pipelines.
- Robustness: Models often learn statistical shortcuts (dataset biases) rather than the underlying logic of a task.
To address these, the authors advocate for "learning from task descriptions," where a model is "programmed" via a natural language instruction ($d_\tau$) to process an input ($X$) and produce an output ($Y$).
The ZEST Benchmark
To rigorously test this paradigm, the authors developed ZEST, a dataset containing 1,251 task descriptions across three domains: U.S. Presidents, dog breeds, and U.S. National Parks. Unlike standard datasets, ZEST is structured to test specific types of generalization.
As shown in Table 1, the benchmark includes:
- Paraphrases: Testing invariance to wording.
- Semantic Flips: Changing a single keyword to ensure the model isn't just matching patterns.
- Composition: Combining tasks using logical operators (e.g., "and", "or").
- Output Structure: Requiring formatted results like JSON.
Methodology: Redefining Success with Task Competence
The authors frame task descriptions as questions paired with context paragraphs, allowing them to utilize pre-trained text-to-text models like T5 and BART. However, they argue that standard average accuracy metrics are insufficient for instruction following.
They introduce Task Competence (C@T), a metric that measures the percentage of entire tasks a model solves correctly. For a task to be considered "solved" at a threshold $T$, the model must achieve at least $T\%$ F1 score across all inputs for that specific task. This prevents models from appearing successful by getting easy examples right while failing to understand the core instruction.
Results and the Competence Gap
The experiments utilized the T5-11B model, which was further enhanced through multi-task learning on other QA datasets. Despite its scale, the model struggled significantly with the ZEST benchmark.
As seen in Table 2, while the model achieves a mean F1 of 56, its Task Competence at the 90% threshold (C@90) is only 12%. In contrast, humans achieve 42% under the same conditions. The ablation studies ("Question only" and "Context only") confirm that the dataset is robust; the model cannot "cheat" by looking at the context alone, as it achieves near-zero scores without the task description.
Discussion: The Path Forward
The results highlight several frontiers for general-purpose AI. Models particularly struggle with Composition and Output Structure, indicating a lack of systematic reasoning and an inability to adhere to strict formatting constraints in a zero-shot setting.
The authors suggest that while zero-shot instruction following is the ultimate goal, a "few-shot" approach—providing a description plus a few examples—may serve as a practical middle ground. Ultimately, closing the 30-point gap between AI and human performance on ZEST remains a major challenge for the field.
Summary updated