Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learning

Fan Yin, Jesse Vig, Philippe Laban, Shafiq Joty, Caiming Xiong, Chien-Sheng WuACL 2023 · DOI

Introduction

Large Language Models (LLMs) are frequently celebrated for their ability to follow human instructions, a capability largely attributed to "instruction learning" on datasets like Super-NaturalInstructions (NIv2). These datasets pair hundreds of diverse tasks with detailed, human-written definitions. However, creating these definitions is an expensive and inconsistent process. This research investigates a fundamental question: Do models actually "read" and comprehend these instructions, or are they merely identifying patterns despite the text?

By systematically analyzing and pruning task definitions, this paper reveals that LLMs actually ignore approximately 60% of the text in a typical instruction. Furthermore, the study demonstrates that replacing messy human prose with a structured format—combined with a specialized "meta-tuning" phase—can significantly enhance model performance while using far less text.

The Utility of Task Definitions

To understand which parts of an instruction actually drive performance, the researchers employed two primary methods of investigation: human-informed ablation and automated compression.

Human-Informed Ablation

The researchers categorized every sentence in the NIv2 task definitions into eight distinct types, such as Input Content, Action Content, and Label List. By systematically removing these categories, they identified what "breaks" the model. The findings were striking:

  • Label Information is King: For classification tasks, the "Label List" and "Label Definitions" are the most critical components. Removing the description of the output space causes performance to crater.
  • Input Descriptions are Often Fluff: Telling a model "You will be given a sentence..." is almost entirely useless. Removing descriptions of the input format caused little to no performance degradation.

Syntax-guided Task Definition Compression (STDC)

The team developed the STDC algorithm, a black-box tool that uses constituency parse trees to iteratively prune phrases from a definition. If removing a phrase did not hurt performance, it was discarded. The results showed that models are "lazy" readers; the algorithm found that models could lose 60% of their instruction tokens and actually see an improvement in performance. This suggests that long, flowery human instructions often introduce noise that distracts the model from the core task.

Proposed Solution: Structured Reformulation and Meta-Tuning

Based on the insight that LLMs behave more like "engineers" than "poets," the researchers proposed moving away from natural language prose in favor of a structured JSON-like triplet format:

  1. Task Input: What is the model looking at?
  2. Task Action: What should it do?
  3. Task Output: What should the result look like?

To help the model transition to this format, the authors introduced a Meta-Tuning stage. In this phase, the model learns to associate demonstration examples with these specific triplet components. This prepares the model to better utilize the structured instructions during the final instruction learning phase.

Experimental Results

The study evaluated BART-Large (400M), T5-Large (770M), and T5-XL (3B) on the English portion of NIv2, testing on 119 unseen tasks across 12 categories.

  • Compression Gains: Using the STDC algorithm to prune instructions led to a ~3 point Rouge-L improvement for T5-XL, despite removing more than half the text.
  • The Triplet Advantage: Replacing full definitions with the structured (Input, Action, Output) triplets improved performance across the board.
  • The Winning Combination: Combining the triplet format with the meta-tuning stage yielded a 4.2 Rouge-L improvement for BART-Large and a 2.1 improvement for T5-XL over the standard state-of-the-art instruction learning approach.

Conclusion and Future Directions

This research challenges the assumption that more detailed human instructions lead to better model performance. Instead, it suggests that models prioritize structure and specific output constraints over descriptive prose. While the study focused on traditional NLP tasks (classification, summarization) in English, it opens the door for rethinking how we communicate tasks to frontier models.

Future work remains to determine if these findings hold for open-ended generation or complex reasoning tasks, and whether the behavior of much larger models (e.g., GPT-4) mirrors the "lazy reading" observed in the T5 and BART families. For now, the evidence suggests that when it comes to instructions, less is often more—provided that "less" is highly structured.

Summary updated

BibTeX
@inproceedings{yin2023did,
  author = {Fan Yin and Jesse Vig and Philippe Laban and Shafiq Joty and Caiming Xiong and Chien-Sheng Wu},
  title = {{Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.172},
}

Loading related content...

Figure