TL;DR
Instruction-tuned language models are surprisingly brittle: changing the phrasing of a task—even if the meaning remains identical—can lead to significant performance drops. This paper quantifies this "phrasing sensitivity" and introduces a lightweight "soft prompt" alignment method that forces models to treat semantically equivalent instructions similarly, significantly boosting their zero-shot robustness.
What Problem Does This Paper Address?
Instruction tuning (e.g., Flan, Alpaca) has been a breakthrough for Large Language Models (LLMs), allowing relatively small models to perform complex tasks just by following natural language directions. The goal is "zero-shot" generalization: the model should understand a task it has never seen before simply by reading the instruction.
However, there is a hidden catch. Most instruction-tuned models are trained on a fixed set of templates. If a user provides an instruction that is semantically correct but phrased differently than the training templates, the model often fails. This paper investigates exactly how sensitive these models are to natural language variation and asks: can we make them more robust to the "way" a user asks a question?
Methods
The researchers took a two-pronged approach:
- Robustness Evaluation: They recruited 36 NLP graduate students to write 319 novel instructions for over 80 tasks. They then compared how models like Flan-T5, Alpaca, and T0 performed when given these "unobserved" instructions versus the "observed" instructions used during their original training.
- Instruction Alignment (The Fix): To fix this sensitivity, they proposed a method called Soft Prompt Alignment. Instead of retraining the whole model, they added a small set of learnable "soft prompt" parameters (prefix tokens) to the input. They trained these parameters using a special loss function (KL-divergence) that encourages the model to produce the same output probability distribution regardless of whether it receives a standard reference instruction or a paraphrased version.
Datasets
The study evaluated models across two major benchmarks:
- MMLU (Massive Multitask Language Understanding): 57 tasks requiring expert knowledge across various domains.
- BIG-BENCH LITE (BBL): A subset of 18 diverse tasks including logic, coding, and common sense.
- The New Instruction Set: The authors introduced a collection of 319 manually written instructions composed by NLP practitioners to serve as a "real-world" test of phrasing variation.
Key Insights and Ideas
- The Phrasing Tax: Using novel (unobserved) instructions consistently degrades performance. For example, Flan-T5 XXL saw an average absolute drop of 6.9 points when moving from seen to unseen phrasings.
- The "Incorrect but Observed" Paradox: In one of the paper's most striking findings, the authors discovered that models often perform better when given an incorrect instruction that they saw during training (e.g., a classification instruction for the wrong task) than when given a correct instruction that is newly phrased. This suggests models are over-fitting to the "shape" of training templates rather than truly "understanding" the task.
- Scale is Not a Cure: Increasing the number of parameters (from 80M to 11B) does not automatically make a model more robust to phrasing shifts. The gap between observed and unobserved instruction performance remains relatively constant as models grow.
Key Results
The authors demonstrated that their Soft Prompt Alignment method (PT+KL) consistently recovers lost performance:
- Flan-T5-3B: Improved performance on unobserved instructions in BBL by 1.8 points compared to the base model.
- Alpaca-7B: Showed a significant 3.7 point improvement on unobserved instructions in BBL.
- Efficiency: The method is lightweight because it only updates the soft prompt embeddings, leaving the massive underlying LLM parameters frozen.
Limitations and Future Work
- Model Size: The study focused on "mid-sized" models (under 20B parameters). It remains to be seen if massive models like GPT-4 (175B+) exhibit the same level of fragility, though the authors note that instruction tuning is most vital for making smaller models competitive.
- Task Variety: The evaluation was restricted to three task types: multiple-choice QA, binary classification, and multi-class classification. Future work could explore open-ended generation or more complex reasoning tasks.
- Instruction Diversity: While 319 instructions is a significant step up from prior work, it still represents a fraction of the infinite ways humans can phrase requests. Expanding the diversity of the alignment data could further improve robustness.
Summary updated