An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Models

Gantavya Bhatt, Yifang Chen, Arnav Mohanty Das, Jifan Zhang, Sang T. Truong, Stephen Mussmann, Yinglun Zhu, Jeff A. Bilmes, Simon S. Du, Kevin G. Jamieson, Jordan T. Ash, Robert D. Nowak

Abstract

Supervised Finetuning (SFT) is a critical step in aligning Large Language Models (LLMs) to follow human instructions. However, the reliance on high-quality, expert-labeled data presents a significant bottleneck due to the high costs of human annotation. While Active Learning offers a potential solution for label efficiency, its iterative nature is computationally prohibitive for massive models. This paper proposes an Experimental Design (ED) framework that selects the most informative prompts in a single, computationally efficient step. By applying classical statistical selection strategies to high-dimensional LLM embeddings, the framework achieves a 50% reduction in annotation costs while maintaining or exceeding state-of-the-art performance.

Introduction

To transform a raw, pre-trained base model into a functional assistant, researchers employ Supervised Finetuning (SFT) on large-scale instruction datasets. As the field moves toward specialized domains and increasingly complex reasoning tasks, the industry can no longer rely on low-quality, automated labels. Instead, expensive human expertise is required.

The traditional approach to maximizing label efficiency is Active Learning, where a model iteratively selects samples, receives labels, and undergoes retraining. For LLMs with 7B or more parameters, this "train-select-repeat" loop is computationally ruinous. This research introduces an alternative: Experimental Design (ED). Unlike Active Learning, ED selects the entire subset of data to be labeled in one shot based on the initial pre-trained model, making it a practical solution for large-scale model development.

Methodology

The framework evaluates several strategies to identify the most informative prompts from a pool of 99,000 unlabeled instructions from the FLAN V2 dataset. The researchers focused on three primary selection categories:

  1. Uncertainty-Based Selection: This method targets prompts where the model exhibits the highest confusion. Beyond standard metrics like Mean Entropy, the authors introduced Min Margin. This score identifies the specific token in a generated sequence where the model is most conflicted between its top two predictions, capturing the "weakest link" in the model's reasoning.
  2. Diversity-Based Selection (k-Center): This strategy ensures the training data covers the widest possible range of tasks by selecting prompts that are spread out across the model's representation space.
  3. Submodular Selection (Facility Location): This approach optimizes for a subset that is both diverse and representative of the entire data pool. By using mathematical functions like the RBF Kernel or Cosine Similarity, the method selects a "goldilocks" set of examples that effectively "cover" the space of all possible prompts.

Experimental Setup

The study utilized a LLaMA-2 7B model as the base. The effectiveness of the selection strategies was measured against a random sampling baseline across several rigorous benchmarks:

  • MMLU (Massive Multitask Language Understanding): Evaluating factual knowledge across 57 subjects.
  • BBH (Big-Bench-Hard): Testing complex reasoning capabilities.
  • AlpacaEval: A head-to-head comparison where GPT-4 Turbo acts as a judge to evaluate model outputs against human-like preferences.

Key Insights and Results

Efficiency and Performance

The Experimental Design framework demonstrated that more data is not always better. Models trained on only 45,000 prompts selected via the Facility Location strategy performed as well as, or better than, models trained on 90,000 prompts selected at random. This represents a 50% reduction in annotation costs.

Strategy Comparison

While uncertainty-based methods are common, the research found that pure uncertainty can lead to redundancy; a model might be uncertain about many prompts that all represent the same underlying difficulty. The Facility Location strategy consistently emerged as the winner, providing a roughly 2% accuracy boost over random sampling on MMLU and BBH. By ensuring the model sees a diverse yet representative sample of the data, it avoids the pitfalls of redundant or outlier-heavy datasets.

Human-Centric Validation

In the AlpacaEval benchmark, the model trained on the 45k "smartly selected" prompts achieved a 52.8% win rate against a model trained on 90k random samples. This confirms that the Experimental Design framework not only improves benchmark scores but also enhances the model's ability to generate responses preferred by human-like evaluators.

Conclusion and Future Work

This work demonstrates that Experimental Design is a computationally cheap and highly effective alternative to iterative Active Learning for LLMs. By selecting data in a single pass, researchers can significantly cut costs without sacrificing model quality.

Future research directions include scaling these strategies to 70B+ parameter models and investigating the impact of noisy human labels. Additionally, exploring how these selection techniques influence the subsequent Reinforcement Learning from Human Feedback (RLHF) phase could further optimize the entire LLM alignment pipeline.

Summary updated

Abstract

Supervised finetuning (SFT) on instruction datasets has played a crucial role in achieving the remarkable zero-shot generalization capabilities observed in modern large language models (LLMs). However, the annotation efforts required to produce high quality responses for instructions are becoming prohibitively expensive, especially as the number of tasks spanned by instruction datasets continues to increase. Active learning is effective in identifying useful subsets of samples to annotate from an unlabeled pool, but its high computational cost remains a barrier to its widespread applicability in the context of LLMs. To mitigate the annotation cost of SFT and circumvent the computational bottlenecks of active learning, we propose using experimental design. Experimental design techniques select the most informative samples to label, and typically maximize some notion of uncertainty and/or diversity. In our work, we implement a framework that evaluates several existing and novel experimental design techniques and find that these methods consistently yield significant gains in label efficiency with little computational overhead. On generative tasks, to reach the same generalization performance, our methods save 50% of the annotation cost compared to random sampling.
BibTeX
@inproceedings{bhatt2024experimental,
  author = {Gantavya Bhatt and Yifang Chen and Arnav Mohanty Das and Jifan Zhang and Sang T. Truong and Stephen Mussmann and Yinglun Zhu and Jeff A. Bilmes and Simon S. Du and Kevin G. Jamieson and Jordan T. Ash and Robert D. Nowak},
  title = {{An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.390},
}
Figure