OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization

Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, Xian Li, Brian O'Horo, Gabriel Pereyra, Jeff Wang, Christopher Dewan, Asli Celikyilmaz, Luke Zettlemoyer, Ves Stoyanov

TL;DR

This paper introduces OPT-IML, a novel instruction-tuned language model that significantly outperforms its base OPT model across various NLP tasks. By developing a comprehensive benchmark and systematically analyzing instruction-tuning decisions, the authors provide crucial insights into scaling these models for improved generalization.

What Problem Does This Paper Address?

Recent advancements in large language models (LLMs) have shown that instruction-tuning—fine-tuning LLMs on tasks described via instructions—greatly improves their ability to generalize to unseen tasks in zero-shot and few-shot settings. However, there's a lack of understanding regarding the impact of various design choices in the instruction-tuning process. These choices include the scale and diversity of the instruction-tuning benchmark, task sampling strategies, the inclusion of specialized datasets (e.g., for reasoning and dialogue), and the fine-tuning objectives themselves. This paper aims to address this gap by systematically characterizing these trade-offs and establishing best practices for large-scale instruction-tuning of LLMs.

Methods

The core of this work lies in developing a comprehensive framework for instruction meta-learning (IML) and generalization evaluation. The authors created OPT-IML Bench, a large benchmark comprising 1,991 NLP tasks consolidated from 8 existing benchmarks and grouped into over 100 task categories. This benchmark is used to evaluate models across three generalization levels:

  1. Fully held-out categories: Tasks from categories not seen during training.
  2. Partially held-out categories: Unseen tasks from categories seen during training.
  3. Fully supervised: New instances from tasks seen during training.

They then systematically analyzed the effects of various instruction-tuning decisions, including:

  • Dataset and benchmark sampling strategies: Investigating how different task proportions affect performance.
  • Scaling laws: Examining the impact of increasing the number of tasks and categories.
  • Specialized datasets: Evaluating the benefit of incorporating reasoning and dialogue datasets.
  • Demonstrations: Studying the effect of including examples in the prompt.

Based on these insights, they fine-tuned OPT models (30B and 175B parameters) using the OPT-IML Bench. The fine-tuning objective used is a next-word prediction objective conditioned on source tokens (instructions and inputs) and only including loss terms from target tokens (labels).

Datasets

The paper introduces OPT-IML Bench, a new benchmark for instruction meta-learning. It consolidates tasks from 8 existing benchmarks:

  • Super-NaturalInstructions
  • PromptSource
  • CrossFit
  • FLAN
  • ExMix
  • T5
  • UnifiedSKG
  • Reasoning

This benchmark contains over 1,991 NLP tasks, categorized into over 100 task clusters. The paper also mentions using datasets related to pre-training, reasoning, and dialogue for fine-tuning.

Key Insights and Ideas

  • Scaling is Crucial: Instruction-tuning benefits significantly from scaling up the number of tasks and categories, especially for fully held-out and partially supervised generalization.
  • Task Diversity Matters: While increasing the number of tasks is beneficial, the diversity of task formats and instruction styles is also important. Biasing the model towards specific formats can hinder generalization.
  • Reasoning Datasets Boost Performance: Incorporating reasoning datasets, even in small proportions, significantly improves performance on reasoning tasks and other held-out categories.
  • Pre-training Data Stabilizes Training: Using a small percentage of pre-training data during fine-tuning can make the training process more stable and improve performance, up to a certain threshold.
  • Dialogue Data Can Be Detrimental: Adding dialogue data, surprisingly, led to performance degradation on certain tasks, suggesting that it might not be universally beneficial and can even harm performance on specific task types.
  • Meta-Training for In-Context Learning is Tricky: While meta-training with demonstrations can improve in-context learning, it can also lead to performance degradation on generation tasks and requires careful handling of prompt lengths and separators.

Key Results

  • OPT-IML outperforms OPT: The instruction-tuned OPT-IML models (30B and 175B) significantly outperform their base OPT counterparts on various benchmarks, showing improvements of 6-7% on average for 0-shot accuracy.
  • Competitive Performance: OPT-IML models are competitive with existing instruction-tuned models that were fine-tuned on individual benchmarks.
  • Improved Generalization: OPT-IML demonstrates strong performance across all three generalization levels (fully held-out, partially supervised, and fully supervised).
  • Scaling Effects: Experiments showed that increasing the number of training tasks and categories generally leads to better performance, especially for generalization to unseen tasks.
  • Reasoning Data Benefits: Adding 1% of reasoning data led to substantial performance gains on reasoning tasks and other held-out categories.

Limitations and Future Work

  • Interactions Between Factors: The paper acknowledges that the studied factors (e.g., task mixing, dataset proportions) might interact, and the optimal settings found for 30B models might not directly translate to larger scales.
  • Category Assignment Subjectivity: The assignment of tasks to categories can be subjective, and different category assignments might lead to different optimal tuning strategies.
  • Responsible AI Concerns: While OPT-IML models show improved performance, they are still susceptible to risks associated with LLMs, such as factual inaccuracies, toxic language generation, and enforcing stereotypes. The authors emphasize the need for responsible deployment.
  • Future Work: The authors suggest exploring continuous learning to adapt models to new data and domains, as well as further investigating the interactions between different instruction-tuning variables. They also highlight the potential for further research into improving reasoning capabilities and understanding the impact of different prompt formats.

Summary updated

Abstract

Recent work has shown that fine-tuning large pre-trained language models on a collection of tasks described via instructions, a.k.a. instruction-tuning, improves their zero and few-shot generalization to unseen tasks. However, there is a limited understanding of the performance trade-offs of different decisions made during the instruction-tuning process. These decisions include the scale and diversity of the instruction-tuning benchmark, different task sampling strategies, fine-tuning with and without demonstrations, training using specialized datasets for reasoning and dialogue, and finally, the fine-tuning objectives themselves. In this paper, we characterize the effect of instruction-tuning decisions on downstream task performance when scaling both model and benchmark sizes. To this end, we create OPT-IML Bench: a large benchmark for Instruction Meta-Learning (IML) of 2000 NLP tasks consolidated into task categories from 8 existing benchmarks, and prepare an evaluation framework to measure three types of model generalizations: to tasks from fully held-out categories, to held-out tasks from seen categories, and to held-out instances from seen tasks. Through the lens of this framework, we first present insights about instruction-tuning decisions as applied to OPT-30B and further exploit these insights to train OPT-IML 30B and 175B, which are instruction-tuned versions of OPT. OPT-IML demonstrates all three generalization abilities at both scales on four different evaluation benchmarks with diverse tasks and input formats -- PromptSource, FLAN, Super-NaturalInstructions, and UnifiedSKG. Not only does it significantly outperform OPT on all benchmarks but is also highly competitive with existing models fine-tuned on each specific benchmark. We release OPT-IML at both scales, together with the OPT-IML Bench evaluation framework.
BibTeX
@misc{iyer2022opt,
  author = {Srinivasan Iyer and Xi Victoria Lin and Ramakanth Pasunuru and Todor Mihaylov and Daniel Simig and Ping Yu and Kurt Shuster and Tianlu Wang and Qing Liu and Punit Singh Koura and Xian Li and Brian O'Horo and Gabriel Pereyra and Jeff Wang and Christopher Dewan and Asli Celikyilmaz and Luke Zettlemoyer and Ves Stoyanov},
  title = {{OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization}},
  year = {2022},
  eprint = {2212.12017},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.12017},
}
Figure