FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain

Introduction

Supervised Fine-Tuning (SFT) is a critical step in adapting pre-trained Large Language Models (LLMs) to specific downstream tasks. However, as models scale to billions or trillions of parameters, the computational cost of training on massive datasets becomes a significant bottleneck. While parameter-efficient methods like LoRA reduce the number of trainable weights, the volume of training data remains a primary driver of cost and time.

Current data selection strategies often rely on random sampling, diversity-based "coresets," or quality-based filtering. These methods, however, do not explicitly optimize for the information gain relative to the model's parameters. FisherSFT addresses this by asking: given a fixed budget of $n$ samples, which subset provides the most information to the model?

Methodology: Optimal Experimental Design

FisherSFT is grounded in the theory of Optimal Experimental Design. The goal is to select a subset of data that maximizes the Fisher Information Matrix (FIM), which represents the amount of information the data carries about the model parameters.

The Information Objective

The core objective is to maximize the log-determinant of the Hessian of the LLM's log-likelihood. Because calculating the full Hessian for an LLM is computationally prohibitive due to the massive parameter space, the authors employ a linearization at the last layer. By treating the final layer as a multinomial logistic regression and using pre-logit embeddings as fixed features, the problem becomes tractable.

Greedy Selection Algorithm

The "log-det" objective is submodular, meaning it exhibits diminishing returns. This property allows for the use of a greedy algorithm to select sentences sequentially.

Expand

Source, p. 3

To further enhance efficiency, the authors developed a "Fast Greedy" variant that utilizes caching and batching, providing an $O(B)$ speedup suitable for large-scale datasets.

Experimental Results

The effectiveness of FisherSFT was evaluated across synthetic data, word2vec embeddings, and real-world fine-tuning of GPT-2 on Shakespearean text.

Sample Efficiency and Error Reduction

In synthetic and word2vec environments, FisherSFT consistently outperformed baselines such as Uniform Sampling, Density Sampling, and Clustered Data Selection. Notably, FisherSFT achieved error rates with 1,000 samples that other methods required 2,000 samples to match.

Expand

Source, p. 7

Generative Quality

For the GPT-2 fine-tuning task, outputs were evaluated by a "GPT-4o judge." FisherSFT-selected data resulted in text that was rated as more fluent and stylistically accurate than data selected via random sampling or other state-of-the-art methods. Preference scores showed that FisherSFT was preferred over uniform sampling up to 80% of the time at smaller sample sizes.

Discussion and Limitations

While FisherSFT demonstrates significant gains in data efficiency, several considerations remain:

• Last-Layer Proxy: The method assumes that maximizing information at the final layer serves as an effective proxy for the whole network. While empirical results are strong, this may not capture all learning dynamics in deeper transformer layers.
• Computational Overhead: Although the "Fast" version is efficient, it still requires an initial embedding pass over the candidate dataset.
• Scalability: Future work will involve testing the method on frontier models with much larger embedding dimensions to confirm its scalability.

@inproceedings{deb2025fishersft,
  author = {Rohan Deb and Kiran Koshy Thekumparampil and Kousha Kalantari and Gaurush Hiranandani and Shoham Sabach and Branislav Kveton},
  title = {{FisherSFT: Data-Efficient Supervised Fine-Tuning of Language Models Using Information Gain}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=e02oLEbehE},
}