Instruction Diversity Drives Generalization To Unseen Tasks

Dylan Zhang, Justin Wang, François ChartonarXiv 2024 · OpenReview

Abstract

Instruction tuning—fine-tuning a large language model (LLM) on pairs of instructions and desired outcomes—is an approach that enables pre-trained language models to perform real-world tasks and follow human instructions. Its practical success depends on the model learning a broader set of instructions than those it was trained on. Yet the factors that determine model generalization to such unseen tasks are not well understood. In this paper, we experiment with string rewrites, a symbolic task that serves as a building block for Turing complete Markov algorithms while allowing experimental control of "inputs" and "instructions". We investigate the trade-off between the number of instructions the model is trained on and the number of training samples provided for each instruction and observe that the diversity of the instruction set determines generalization. Generalization emerges once a diverse enough set of tasks is provided, even though very few examples are provided for each task. Instruction diversity also ensures robustness with respect to non-uniform distributions of instructions in the training set.

Introduction

Instruction tuning is the "secret sauce" that transforms raw, pre-trained Large Language Models (LLMs) into helpful assistants like ChatGPT. While its efficacy is well-documented, the underlying mechanics of how these models generalize to tasks they were not explicitly trained on remain poorly understood. In real-world scenarios, instruction-following datasets are often complex, mixing linguistic nuances, world knowledge, and varied formatting, which makes isolating variables difficult. This study addresses this by utilizing symbolic string rewrites based on Markov algorithms. This controlled environment allows for the precise manipulation of the number of unique instructions versus the number of examples per instruction to identify the triggers for generalization.

Methodology

The researchers employed a two-pronged approach to investigate the drivers of generalization:

  1. Symbolic Experiments: Small GPT-2 models (6 layers) were trained from scratch on string-rewriting tasks. A task is defined as a rule $(x \to y)$ applied to a string $z$. Because Markov algorithms are Turing-complete, these simple rules can theoretically represent any computation.
  2. Pre-trained Model Fine-tuning: To validate findings on larger models, Llama-2-7b was fine-tuned using LoRA on a complex "encrypted rewriting" task. This required the model to identify a word in a sentence, replace it, and then apply a Caesar cipher to that word—a multi-step reasoning task.
The encrypted rewriting task requires the model to perform both string replacement and a Caesar cipher encryption, testing multi-step instruction following.

The key experimental lever was the trade-off between $I$ (number of unique instructions) and $S$ (number of examples per instruction), while keeping the total training budget constant.

The Diversity Threshold and Phase Transition

The most striking finding is a "sharp phase transition" in model performance. In the basic symbolic setup, models trained on fewer than 300 unique instructions failed to generalize entirely. However, once the diversity reached approximately 400 instructions, accuracy skyrocketed toward near-perfection.

Average accuracy shows a sharp phase transition as the number of unique instructions increases, even as the number of examples per instruction decreases.

This result demonstrates that instruction diversity is significantly more important than the volume of training samples per task. For a fixed data budget, it is better to have 1,000 different instructions with 10 examples each than 10 instructions with 1,000 examples each. The latter leads to rote memorization, while the former forces the model to learn the fundamental mechanism of instruction following.

Semantic Diversity and Robustness

It is not just the quantity of instructions that matters, but their semantic variety. Training on rules involving only specific patterns (e.g., repeated characters) does not enable generalization to different patterns (e.g., mirroring). Generalization to "unconstrained" rules only occurs when the training set includes a diverse mix of semantic rule types.

Furthermore, the models proved robust to "long-tail" distributions. In the real world, some tasks are common while others are rare (a power-law distribution). The authors found that if the total number of unique instructions is high enough (e.g., 10,000), the model remains highly accurate even if the distribution of examples is extremely uneven.

Conclusion

This research provides strong empirical evidence that instruction diversity is the primary driver for generalization in LLMs. The discovery of a sharp phase transition suggests a minimum threshold of variety is required for a model to internalize the logic of instruction following. While symbolic tasks lack the knowledge retrieval aspects of natural language, they provide a clear foundation for optimizing instruction-tuning datasets, suggesting that breadth of tasks should be prioritized over depth of examples per task.

Summary updated

Abstract

Instruction tuning -- fine-tuning a large language model (LLM) on pairs of instructions and desired outcomes -- is an approach that enables pre-trained language models to perform real-world tasks and follow human instructions. Its practical success depends on the model learning a broader set of instructions than those it was trained on. Yet the factors that determine model generalization to such \emph{unseen tasks} are not well understood. %To understand the driving factors of generalization, In this paper, we experiment with string rewrites, a symbolic task that serves as a building block for Turing complete Markov algorithms while allowing experimental control of "inputs" and "instructions". We investigate the trade-off between the number of instructions the model is trained on and the number of training samples provided for each instruction and observe that the diversity of the instruction set determines generalization. Generalization emerges once a diverse enough set of tasks is provided, even though very few examples are provided for each task. Instruction diversity also ensures robustness with respect to non-uniform distributions of instructions in the training set.
BibTeX
@misc{zhang2024instruction,
  author = {Dylan Zhang and Justin Wang and François Charton},
  title = {{Instruction Diversity Drives Generalization To Unseen Tasks}},
  year = {2024},
  eprint = {2402.10891},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.10891},
}
Figure