60 Data Points are Sufficient to Fine-Tune LLMs for Question-Answering

Junjie Ye, Yuming Yang, Qi Zhang, Tao Gui, Xuanjing Huang, Peng Wang, Zhongchao Shi, Jianping Fan

TL;DR

Supervised fine-tuning (SFT) for question-answering tasks doesn't require massive datasets; as few as 60 carefully selected data points can successfully activate an LLM's pre-existing world knowledge. However, the choice of data matters immensely: fine-tuning a model on facts it barely memorized during pre-training can severely damage its ability to recall things it actually knows well.

What Problem Does This Paper Address?

When we fine-tune Large Language Models (LLMs) using Supervised Fine-Tuning (SFT), the common assumption is often "the more data, the better." However, for factual question-answering (QA) tasks, the core knowledge is already baked into the model's parameters during its massive pre-training phase. SFT's primary job isn't to teach the model new facts, but rather to teach it how to behave like a QA system.

Prior attempts to optimize SFT data selection suffered from two major flaws: they evaluated what a model "knows" using highly volatile few-shot prompts, and they categorized model memory into overly coarse buckets (e.g., just low, medium, and high). This paper introduces a highly granular approach to answer three pivotal questions:

  1. How much data do we actually need to unlock QA capabilities?
  2. How do different levels of pre-trained data memorization affect post-SFT performance?
  3. How do these data requirements vary across different model families?

Methods

To accurately gauge what a model truly "remembers" before fine-tuning, the authors developed a robust Multi-Template Complementation Mechanism.

Instead of relying on a single question format, they break down knowledge into factual triples (subject, relation, object) and generate 21 distinct text-completion templates for each fact. For example, to test if a model knows where "Painblanc" is, they query it with variations like "Painblanc is located in..." and "The country of Painblanc is..."

An example of the multi-template complementation mechanism used to evaluate model memory.

By sampling the model's completions 10 times per template (at a temperature of 0.7), they calculate a precise memory score between 0 and 1. They then segment the dataset into five distinct memory levels ($D_{train-0}$ to $D_{train-4}$), ranging from completely un-memorized to deeply memorized facts. They apply this methodology across four popular open-source models: LLaMA-2-7B, LLaMA-2-13B, LLaMA-3-8B, and Qwen-2-7B.

Datasets

The authors utilized ENTITYQUESTIONS, a benchmark dataset containing Wikipedia-sourced factual knowledge across 24 distinct topics. They split the dataset into:

  • In-Domain Data ($D_{train}$ / $D_{test}$): 12 location-related topics (e.g., birthplaces, headquarters) comprising 92,435 training samples and 11,546 test samples.
  • Out-of-Domain Data ($D_{test-ood}$): 12 entirely unrelated topics (e.g., authors, musical styles) comprising 9,660 samples to test the models' generalization capabilities.

Key Insights and Ideas

  • The 60-Sample Sweet Spot: Across all tested models, scaling up the number of SFT training examples beyond 60 did not yield significant performance gains. In fact, flooding the model with thousands of examples frequently degraded its performance. SFT acts as a lightweight formatting trigger; once the model learns the QA format, extra data just risks overfitting.
  • The "Diagonal Phenomenon": The authors discovered that training an LLM on data from a specific memory level boosts its performance on that specific level of knowledge the most.
  • The Danger of Mismatched Data: Fine-tuning a model on facts it failed to memorize during pre-training ($D_{train-0}$) acts as a poison pill. It severely disrupts the model's internal representations, drastically impairing its ability to answer questions about topics it previously knew perfectly ($D_{test-4}$).
  • Unique Knowledge Profiles: Different model families possess vastly different internal knowledge distributions due to their distinct pre-training corpora. As a result, a static SFT dataset that works wonders for LLaMA-3 might cause a performance cliff in Qwen-2.
In-domain performance of various LLMs across different SFT data sizes and memory levels. Notice how performance frequently peaks at just 60 samples.

Key Results

  • Minimal Data, Maximum Impact: As shown in the empirical evaluations, models trained on just 60 samples achieved optimal or near-optimal accuracy on in-domain tasks.
  • Knowledge Distribution Variance: Pairwise heatmaps revealed stark differences in what models remember. For instance, facts completely un-memorized by Qwen-2-7B were frequently found to be deeply memorized by LLaMA-3-8B.
Heatmaps illustrating the massive discrepancies in pre-trained knowledge distribution between different LLM families.
  • The Cost of Blind Data Sharing: When the exact same 60-sample dataset ($D^*_{train}$) was used to fine-tune different models, LLaMA-3-8B generalized well, but Qwen-2-7B suffered a massive 13.42% drop in accuracy compared to when it was fine-tuned on a custom dataset tailored to its own memory profile.

Limitations and Future Work

While this paper provides compelling empirical proof that "less is more" for SFT data volume, it remains primarily an observational study. The authors acknowledge that they have not yet uncovered the exact mechanical reasons why training on un-memorized data actively damages high-memory knowledge pathways.

The team plans to follow up this work with mechanistic interpretability studies, looking directly at internal weight changes and attention shifts during the SFT process to explain these phenomena.

Summary updated

Abstract

Large language models (LLMs) encode extensive world knowledge through pre-training on massive datasets, which can then be fine-tuned for the question-answering (QA) task. However, effective strategies for fine-tuning LLMs for the QA task remain largely unexplored. To address this gap, we categorize supervised fine-tuning (SFT) data based on the extent of knowledge memorized by the pretrained LLMs and conduct a series of empirical analyses. Our experiments, involving four LLMs from three different model families, focus on three key factors: the amount of data required for SFT, the impact of different SFT datasets on model performance, and how data requirements vary across LLMs. The results show that as few as 60 data points during the SFT stage can activate the knowledge encoded during pre-training, enabling LLMs to perform the QA task. Additionally, SFT with data of varying memory levels has a significant impact on LLM performance, with the optimal dataset differing based on the specific model being fine-tuned. Future research will delve deeper into the mechanisms underlying these phenomena.
BibTeX
@misc{ye2024data,
  author = {Junjie Ye and Yuming Yang and Qi Zhang and Tao Gui and Xuanjing Huang and Peng Wang and Zhongchao Shi and Jianping Fan},
  title = {{60 Data Points are Sufficient to Fine-Tune LLMs for Question-Answering}},
  year = {2024},
  eprint = {2409.15825},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.15825},
}
Figure