Evaluating the Zero-shot Robustness of Instruction-tuned Language Models

Jiuding Sun, Chantal Shaib, Byron C. Wallace

TL;DR

Instruction-tuned language models are surprisingly brittle: changing the phrasing of a task—even if the meaning remains identical—can lead to significant performance drops. This paper quantifies this "phrasing sensitivity" and introduces a lightweight "soft prompt" alignment method that forces models to treat semantically equivalent instructions similarly, significantly boosting their zero-shot robustness.

What Problem Does This Paper Address?

Instruction tuning (e.g., Flan, Alpaca) has been a breakthrough for Large Language Models (LLMs), allowing relatively small models to perform complex tasks just by following natural language directions. The goal is "zero-shot" generalization: the model should understand a task it has never seen before simply by reading the instruction.

However, there is a hidden catch. Most instruction-tuned models are trained on a fixed set of templates. If a user provides an instruction that is semantically correct but phrased differently than the training templates, the model often fails. This paper investigates exactly how sensitive these models are to natural language variation and asks: can we make them more robust to the "way" a user asks a question?

Figure 1: How well do models trained on instruction-tuning datasets generalize to novel instructions (unobserved in training)? Our analysis suggests that they do not do so very well. Above we show a case where pairing an example with an observed instruction yields the correct output, while providing a distinct but semantically equivalent instruction produces an incorrect response. We propose and evaluate a simple method that improves this.

Methods

The researchers took a two-pronged approach:

  1. Robustness Evaluation: They recruited 36 NLP graduate students to write 319 novel instructions for over 80 tasks. They then compared how models like Flan-T5, Alpaca, and T0 performed when given these "unobserved" instructions versus the "observed" instructions used during their original training.
  2. Instruction Alignment (The Fix): To fix this sensitivity, they proposed a method called Soft Prompt Alignment. Instead of retraining the whole model, they added a small set of learnable "soft prompt" parameters (prefix tokens) to the input. They trained these parameters using a special loss function (KL-divergence) that encourages the model to produce the same output probability distribution regardless of whether it receives a standard reference instruction or a paraphrased version.

Datasets

The study evaluated models across two major benchmarks:

  • MMLU (Massive Multitask Language Understanding): 57 tasks requiring expert knowledge across various domains.
  • BIG-BENCH LITE (BBL): A subset of 18 diverse tasks including logic, coding, and common sense.
  • The New Instruction Set: The authors introduced a collection of 319 manually written instructions composed by NLP practitioners to serve as a "real-world" test of phrasing variation.

Key Insights and Ideas

  • The Phrasing Tax: Using novel (unobserved) instructions consistently degrades performance. For example, Flan-T5 XXL saw an average absolute drop of 6.9 points when moving from seen to unseen phrasings.
  • The "Incorrect but Observed" Paradox: In one of the paper's most striking findings, the authors discovered that models often perform better when given an incorrect instruction that they saw during training (e.g., a classification instruction for the wrong task) than when given a correct instruction that is newly phrased. This suggests models are over-fitting to the "shape" of training templates rather than truly "understanding" the task.
  • Scale is Not a Cure: Increasing the number of parameters (from 80M to 11B) does not automatically make a model more robust to phrasing shifts. The gap between observed and unobserved instruction performance remains relatively constant as models grow.
Figure 3: Incorrect but observed instructions perform better on average than correct but unobserved instructions. We report averages over benchmarks, but show example instructions on the right for a specific, illustrative task. We provide all instructions in the Appendix.

Key Results

The authors demonstrated that their Soft Prompt Alignment method (PT+KL) consistently recovers lost performance:

  • Flan-T5-3B: Improved performance on unobserved instructions in BBL by 1.8 points compared to the base model.
  • Alpaca-7B: Showed a significant 3.7 point improvement on unobserved instructions in BBL.
  • Efficiency: The method is lightweight because it only updates the soft prompt embeddings, leaving the massive underlying LLM parameters frozen.
Figure 2: Using novel but valid instructions at test time (phrasings unobserved in training) consistently degrades the performance of instruction-tuned LLMs (a). Scale does not necessarily fix this (b).

Limitations and Future Work

  • Model Size: The study focused on "mid-sized" models (under 20B parameters). It remains to be seen if massive models like GPT-4 (175B+) exhibit the same level of fragility, though the authors note that instruction tuning is most vital for making smaller models competitive.
  • Task Variety: The evaluation was restricted to three task types: multiple-choice QA, binary classification, and multi-class classification. Future work could explore open-ended generation or more complex reasoning tasks.
  • Instruction Diversity: While 319 instructions is a significant step up from prior work, it still represents a fraction of the infinite ways humans can phrase requests. Expanding the diversity of the alignment data could further improve robustness.

Summary updated

Abstract

Instruction fine-tuning has recently emerged as a promising approach for improving the zero-shot capabilities of Large Language Models (LLMs) on new tasks. This technique has shown particular strength in improving the performance of modestly sized LLMs, sometimes inducing performance competitive with much larger model variants. In this paper we ask two questions: (1) How sensitive are instruction-tuned models to the particular phrasings of instructions, and, (2) How can we make them more robust to such natural language variation? To answer the former, we collect a set of 319 instructions manually written by NLP practitioners for over 80 unique tasks included in widely used benchmarks, and we evaluate the variance and average performance of these instructions as compared to instruction phrasings observed during instruction fine-tuning. We find that using novel (unobserved) but appropriate instruction phrasings consistently degrades model performance, sometimes substantially so. Further, such natural instructions yield a wide variance in downstream performance, despite their semantic equivalence. Put another way, instruction-tuned models are not especially robust to instruction re-phrasings. We propose a simple method to mitigate this issue by introducing ``soft prompt'' embedding parameters and optimizing these to maximize the similarity between representations of semantically equivalent instructions. We show that this method consistently improves the robustness of instruction-tuned models.
BibTeX
@inproceedings{sun2024evaluating,
  author = {Jiuding Sun and Chantal Shaib and Byron C. Wallace},
  title = {{Evaluating the Zero-shot Robustness of Instruction-tuned Language Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=g9diuvxN6D},
}
Figure