Two-stage LLM Fine-tuning with Less Specialization and More Generalization

Yihan Wang, Si Si, Daliang Li, Michal Lukasik, Felix Yu, Cho-Jui Hsieh, Inderjit S Dhillon, Sanjiv Kumar

TL;DR

Fine-tuning large language models (LLMs) on specific tasks often destroys their general "in-context learning" abilities, a phenomenon the authors attribute to format specialization. To fix this, they introduce ProMoT (Prompt Tuning with MOdel Tuning), a two-stage framework that offloads format learning to a soft prompt before updating the model's weights, preserving versatility without sacrificing task-specific accuracy.

What Problem Does This Paper Address?

LLMs are prized for their ability to solve diverse tasks via few-shot prompting (in-context learning). However, to squeeze out maximum performance for a specific application, researchers often fine-tune them on specialized datasets.

The catch? Standard fine-tuning usually causes the model to "forget" its general-purpose capabilities. For example, if you fine-tune a model on a binary classification task like Natural Language Inference (NLI), it quickly learns to output only "True" or "False." If you then ask that same model to summarize a paragraph, it might stubbornly respond with "True" because it has over-specialized to the NLI format.

The authors identify that this loss of generalization happens incredibly fast—often within the first few hundred steps of training—and is primarily caused by the model overfitting to the format of the data rather than its semantic content.

Figure 1: Loss of in-context learning abili- ties during fine-tuning. We show the learn- ing curve of a model being fine-tuned on RTE dataset while being tested on 1-shot QA datasets. Left axis: Accuracy on RTE. Right axis: Exact match rate on 1-shot QA.

Methods

The authors propose ProMoT (Prompt Tuning with MOdel Tuning), a simple two-stage strategy designed to decouple format learning from semantic learning:

  1. Stage 1: Prompt Tuning: The pretrained model is frozen. A small set of continuous, trainable parameters (a "soft prompt") is prepended to the input and trained on the target task. Because format learning happens first, the soft prompt "absorbs" the task's specific output style and structure.
  2. Stage 2: Model Tuning: The soft prompt learned in Stage 1 is now frozen, and the main model weights are unfrozen and fine-tuned. Since the format is already handled by the prompt, the model's internal weight updates can focus on the deeper semantic nuances of the task.
Figure 4: Overview of ProMoT, our two-stage fine-tuning strategy. We run prompt tuning at Stage 1 and model fine-tuning with the trained prompt at Stage 2. Green denotes trainable parameters and blue means frozen.

At inference time, if the incoming task matches the fine-tuned format, the prompt is kept. If the user wants to perform a different general task (like summarization), the prompt is removed, revealing the model's preserved general-purpose capabilities.

Datasets

The researchers used a variety of datasets to test both specialization and generalization:

  • Fine-tuning Tasks: RTE (Recognizing Textual Entailment), WMT14 En-Fr (Translation), SNLI, and OpenbookQA.
  • Evaluation Tasks (Unseen during tuning):
    • NLI: CB and WiC.
  • Closed-book QA: TriviaQA and WebQuestions.
  • Translation: WMT16 En-Ro and En-De.
  • Summarization: XSum and WikiLingua.

Key Insights and Ideas

  • Format Specialization is the Culprit: The authors found that at the very start of training, the model's gradients are highly aligned with the task format. In Figure 3, they show that the gradient for the actual task is almost identical to the gradient for a "randomized" version of the task where labels are shuffled (meaning the model is just learning the output labels, not the logic).
  • Early Forgetting: General capabilities don't erode slowly; they collapse almost immediately. By step 300 of NLI fine-tuning, 90% of a model's outputs for a completely different QA task become "True" or "False."
  • Semantic Synergy: Interestingly, ProMoT doesn't just preserve general skills; it can enhance them. Fine-tuning on NLI using ProMoT actually improved the model's performance on summarization tasks compared to the base pretrained model, likely because the model learned better "grounding" without being blinded by the NLI format.
Table 1: Qualitative comparison showing how ProMoT preserves summarization ability while standard fine-tuning fails.

Key Results

  • Supervised Performance: ProMoT achieves comparable or even superior results on the target fine-tuning tasks compared to standard fine-tuning. For RTE, ProMoT reached 92.78% accuracy vs. 92.06% for standard tuning.
  • Generalization: On a normalized average of 8 unseen tasks, standard fine-tuning on RTE caused a performance drop of -2.10%. In contrast, ProMoT achieved a gain of +2.58%.
  • Cross-Task Boost: ProMoT trained on English-French translation significantly improved performance on other language pairs (like English-German), whereas standard fine-tuning caused those capabilities to drop to near zero.

Limitations and Future Work

  • Model Scale: Most experiments were conducted on models with fewer than 15 billion parameters (mT5 XXL, PaLM 8b). The authors suggest testing ProMoT on even larger models.
  • Format Absorption: While empirically successful, there is no theoretical guarantee that the soft prompt will absorb all format information in every possible scenario.
  • Computational Cost: ProMoT requires a two-stage training process, which is slightly more complex to manage than a single-stage fine-tuning run.

Overall, ProMoT offers a practical path for engineers who need high-performance specialized models that don't lose their "magic" as general-purpose assistants.

Summary updated

Abstract

Pretrained large language models (LLMs) are general purpose problem solvers applicable to a diverse set of tasks with prompts. They can be further improved towards a specific task by fine-tuning on a specialized dataset. However, fine-tuning usually makes the model narrowly specialized on this dataset with reduced general in-context learning performances, which is undesirable whenever the fine-tuned model needs to handle additional tasks where no fine-tuning data is available. In this work, we first demonstrate that fine-tuning on a single task indeed decreases LLMs' general in-context learning performance. We discover one important cause of such forgetting, format specialization, where the model overfits to the format of the fine-tuned task.We further show that format specialization happens at the very beginning of fine-tuning. To solve this problem, we propose Prompt Tuning with MOdel Tuning (ProMoT), a simple yet effective two-stage fine-tuning framework that reduces format specialization and improves generalization.ProMoT offloads task-specific format learning into additional and removable parameters by first doing prompt tuning and then fine-tuning the model itself with this soft prompt attached. With experiments on several fine-tuning tasks and 8 in-context evaluation tasks, we show that ProMoT achieves comparable performance on fine-tuned tasks to standard fine-tuning, but with much less loss of in-context learning performances across a board range of out-of-domain evaluation tasks. More importantly, ProMoT can even enhance generalization on in-context learning tasks that are semantically related to the fine-tuned task, e.g. ProMoT on En-Fr translation significantly improves performance on other language pairs, and ProMoT on NLI improves performance on summarization. Experiments also show that ProMoT can improve the generalization performance of multi-task training.
BibTeX
@inproceedings{wang2024two,
  author = {Yihan Wang and Si Si and Daliang Li and Michal Lukasik and Felix Yu and Cho-Jui Hsieh and Inderjit S Dhillon and Sanjiv Kumar},
  title = {{Two-stage LLM Fine-tuning with Less Specialization and More Generalization}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=pCEgna6Qco},
}
Figure