Improving Instruct Models for Free: A Study on Partial Adaptation

Ozan Irsoy, Pengxiang Cheng, Jennifer L Chen, Daniel Preotiuc-Pietro, Shiyue Zhang, Duccio PappadopuloEMNLP 2025

TL;DR

Instruct models, while better at following commands, can sometimes lose valuable knowledge from their pre-training. This paper explores "partial adaptation," a technique to fine-tune models less aggressively, finding that it can significantly improve performance on in-context learning tasks without a major hit to instruction following. This suggests a trade-off exists and offers a practical way to balance these abilities.

What Problem Does This Paper Address?

Large Language Models (LLMs) undergo pre-training to learn general knowledge and then post-training (like instruction tuning) to become better at following user instructions. However, this post-training can sometimes lead to "catastrophic forgetting," where the model loses some of its original knowledge, or it might make the model overly verbose, which is detrimental for tasks requiring concise, structured outputs. This verbosity can degrade performance on few-shot, in-context learning (ICL) tasks, where models are expected to learn from a few examples provided in the prompt. The paper aims to investigate the performance trajectory between base (pre-trained) and fully instruction-tuned models, specifically focusing on how to mitigate the negative impacts of aggressive instruction tuning on ICL capabilities.

Methods

The core of this paper's methodology is partial adaptation (PAd). Instead of fully instruction-tuning a base model, PAd creates intermediate models by blending the weights of a base model ($W_B$) and an instruction-tuned model ($W_I$). This is achieved by creating a new model $M_\lambda$ with weights $W_{M_\lambda} = (1-\lambda)W_B + \lambda W_I$, where $\lambda$ is a hyperparameter ranging from 0 (pure base model) to 1 (fully instruct model). By systematically varying $\lambda$, the authors can explore a spectrum of instruction-tuning strengths. This approach is "training-free" in the sense that it doesn't require new training data or gradient updates; it's a post-hoc manipulation of existing model weights.

Datasets

The paper evaluates its approach on a benchmark comprising 21 classic NLP tasks across various domains. These datasets are categorized into:

  • World Knowledge: MMLU, Trivia QA, Natural Questions.
  • Commonsense Reasoning: PIQA, Winogrande, ARC Challenge, HellaSwag.
  • Language Processing and Understanding: Tasks from FLUE (Financial Language Understanding Evaluation) like sentiment analysis, headline classification, and named entity recognition.
  • Symbolic and Logical Problem Solving: BBH (Algo), DROP, TAT-QA, Pacific.
  • Reading Comprehension: SQuAD, QuAC, ConvFinQA.
  • Retrieval-augmented Generation (RAG): Questions from Natural Questions and Trivia QA paired with Wikipedia passages.

The paper does not introduce new datasets but rather curates and utilizes existing ones to create a comprehensive ICL evaluation.

Key Insights and Ideas

  • The "Sweet Spot" for ICL: The most striking finding is that for in-context learning, the optimal performance is almost always achieved with a $\lambda < 1$. This means that partially adapted models, with less aggressive instruction tuning than the fully tuned counterparts, consistently outperform them on ICL tasks.
  • Trade-off is Real: While PAd boosts ICL, it comes at the cost of instruction following ability. The paper quantifies this trade-off, showing that even a small drop in instruction following (e.g., less than 1% on AlpacaEval) can yield significant gains in ICL.
  • Larger Models are More Resilient: For larger models, instruction following performance (as measured by AlpacaEval) shows less sensitivity to $\lambda$ when $\lambda \le 1$. This suggests that for these models, there's a wider range of partial adaptation values that can improve ICL without a substantial degradation in instruction following.
  • PAd as a Post-Training Step: The authors propose that PAd could be a valuable final step in post-training, allowing for a more nuanced balance between instruction following and retained knowledge, especially for tasks where ICL is critical.

Key Results

  • Across 18 open-weight LLMs tested, PAd consistently improved ICL performance over both base and fully instruct models for 17 out of 18 models.
  • The improvement in ICL was often significant, with the largest observed gain being 2.5% on Llama-3 8B.
  • For many models, the optimal $\lambda$ was found to be between 0.5 and 0.6.
  • On AlpacaEval 2.0 (a measure of instruction following), partially adapted models generally showed a slight decrease in performance compared to the fully instruct model, but this decrease was often minimal for values of $\lambda$ close to 1. For instance, a relative win-rate decrease of at most 1% on AlpacaEval 2.0 allowed for a +5.9% relative improvement on ICL for Llama-2 70B.

Limitations and Future Work

Limitations: * The ICL benchmark, while comprehensive with 21 datasets, may not be fully representative of all possible ICL scenarios or model capabilities. * AlpacaEval 2.0, used for instruction following, has a high correlation with Chatbot Arena but might not capture the full spectrum of instruction following performance. * The study is limited to models primarily trained on English data, leaving the generalizability to other languages and multilingual models for future work.

Future Work: * Investigate why PAd improves ICL by examining its impact on different stages of supervised fine-tuning or RL. * Conduct a more thorough comparison of training dynamics during instruction tuning versus the trajectory defined by varying $\lambda$ in PAd, as PAd might replicate full training dynamics. * Explore the application of PAd to models trained on non-English data and multilingual models.

Summary updated

BibTeX
@inproceedings{irsoy2025improving,
  author = {Ozan Irsoy and Pengxiang Cheng and Jennifer L Chen and Daniel Preotiuc-Pietro and Shiyue Zhang and Duccio Pappadopulo},
  title = {{Improving Instruct Models for Free: A Study on Partial Adaptation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure