TL;DR
Supervised fine-tuning (SFT) for question-answering tasks doesn't require massive datasets; as few as 60 carefully selected data points can successfully activate an LLM's pre-existing world knowledge. However, the choice of data matters immensely: fine-tuning a model on facts it barely memorized during pre-training can severely damage its ability to recall things it actually knows well.
What Problem Does This Paper Address?
When we fine-tune Large Language Models (LLMs) using Supervised Fine-Tuning (SFT), the common assumption is often "the more data, the better." However, for factual question-answering (QA) tasks, the core knowledge is already baked into the model's parameters during its massive pre-training phase. SFT's primary job isn't to teach the model new facts, but rather to teach it how to behave like a QA system.
Prior attempts to optimize SFT data selection suffered from two major flaws: they evaluated what a model "knows" using highly volatile few-shot prompts, and they categorized model memory into overly coarse buckets (e.g., just low, medium, and high). This paper introduces a highly granular approach to answer three pivotal questions:
- How much data do we actually need to unlock QA capabilities?
- How do different levels of pre-trained data memorization affect post-SFT performance?
- How do these data requirements vary across different model families?
Methods
To accurately gauge what a model truly "remembers" before fine-tuning, the authors developed a robust Multi-Template Complementation Mechanism.
Instead of relying on a single question format, they break down knowledge into factual triples (subject, relation, object) and generate 21 distinct text-completion templates for each fact. For example, to test if a model knows where "Painblanc" is, they query it with variations like "Painblanc is located in..." and "The country of Painblanc is..."
By sampling the model's completions 10 times per template (at a temperature of 0.7), they calculate a precise memory score between 0 and 1. They then segment the dataset into five distinct memory levels ($D_{train-0}$ to $D_{train-4}$), ranging from completely un-memorized to deeply memorized facts. They apply this methodology across four popular open-source models: LLaMA-2-7B, LLaMA-2-13B, LLaMA-3-8B, and Qwen-2-7B.
Datasets
The authors utilized ENTITYQUESTIONS, a benchmark dataset containing Wikipedia-sourced factual knowledge across 24 distinct topics. They split the dataset into:
- In-Domain Data ($D_{train}$ / $D_{test}$): 12 location-related topics (e.g., birthplaces, headquarters) comprising 92,435 training samples and 11,546 test samples.
- Out-of-Domain Data ($D_{test-ood}$): 12 entirely unrelated topics (e.g., authors, musical styles) comprising 9,660 samples to test the models' generalization capabilities.
Key Insights and Ideas
- The 60-Sample Sweet Spot: Across all tested models, scaling up the number of SFT training examples beyond 60 did not yield significant performance gains. In fact, flooding the model with thousands of examples frequently degraded its performance. SFT acts as a lightweight formatting trigger; once the model learns the QA format, extra data just risks overfitting.
- The "Diagonal Phenomenon": The authors discovered that training an LLM on data from a specific memory level boosts its performance on that specific level of knowledge the most.
- The Danger of Mismatched Data: Fine-tuning a model on facts it failed to memorize during pre-training ($D_{train-0}$) acts as a poison pill. It severely disrupts the model's internal representations, drastically impairing its ability to answer questions about topics it previously knew perfectly ($D_{test-4}$).
- Unique Knowledge Profiles: Different model families possess vastly different internal knowledge distributions due to their distinct pre-training corpora. As a result, a static SFT dataset that works wonders for LLaMA-3 might cause a performance cliff in Qwen-2.
Key Results
- Minimal Data, Maximum Impact: As shown in the empirical evaluations, models trained on just 60 samples achieved optimal or near-optimal accuracy on in-domain tasks.
- Knowledge Distribution Variance: Pairwise heatmaps revealed stark differences in what models remember. For instance, facts completely un-memorized by Qwen-2-7B were frequently found to be deeply memorized by LLaMA-3-8B.
- The Cost of Blind Data Sharing: When the exact same 60-sample dataset ($D^*_{train}$) was used to fine-tune different models, LLaMA-3-8B generalized well, but Qwen-2-7B suffered a massive 13.42% drop in accuracy compared to when it was fine-tuned on a custom dataset tailored to its own memory profile.
Limitations and Future Work
While this paper provides compelling empirical proof that "less is more" for SFT data volume, it remains primarily an observational study. The authors acknowledge that they have not yet uncovered the exact mechanical reasons why training on un-memorized data actively damages high-memory knowledge pathways.
The team plans to follow up this work with mechanistic interpretability studies, looking directly at internal weight changes and attention shifts during the SFT process to explain these phenomena.
Summary updated