Abstract
Pretrained large language models (LLMs) are general purpose problem solvers applicable to a diverse set of tasks with prompts. They can be further improved towards a specific task by fine-tuning on a specialized dataset. However, fine-tuning usually makes the model narrowly specialized on this dataset with reduced general in-context learning performances, which is undesirable whenever the fine-tuned model needs to handle additional tasks where no fine-tuning data is available. In this work, we first demonstrate that fine-tuning on a single task indeed decreases LLMs' general in-context learning performance. We discover one important cause of such forgetting, format specialization, where the model overfits to the format of the fine-tuned task. We further show that format specialization happens at the very beginning of fine-tuning. To solve this problem, we propose Prompt Tuning with MOdel Tuning (ProMoT), a simple yet effective two-stage fine-tuning framework that reduces format specialization and improves generalization. ProMoT offloads task-specific format learning into additional and removable parameters by first doing prompt tuning and then fine-tuning the model itself with this soft prompt attached. With experiments on several fine-tuning tasks and 8 in-context evaluation tasks, we show that ProMoT achieves comparable performance on fine-tuned tasks to standard fine-tuning, but with much less loss of in-context learning performances across a board range of out-of-domain evaluation tasks. More importantly, ProMoT can even enhance generalization on in-context learning tasks that are semantically related to the fine-tuned task, e.g. ProMoT on En-Fr translation significantly improves performance on other language pairs, and ProMoT on NLI improves performance on summarization. Experiments also show that ProMoT can improve the generalization performance of multi-task training.
Introduction: The Challenge of Format Specialization
Pre-trained Large Language Models (LLMs) are celebrated for their "jack-of-all-trades" nature, capable of solving various problems through few-shot in-context learning. However, when developers seek to specialize a model for a specific domain—such as medical diagnosis or legal translation—standard fine-tuning often results in a catastrophic loss of general-purpose intelligence.
This paper identifies a primary driver of this phenomenon: format specialization. This occurs when a model overfits to the specific output structure of the fine-tuning task. For example, a model fine-tuned on a binary classification task (e.g., "True/False") may begin to respond with these labels even when prompted for a summary. Crucially, research shows that this specialization happens almost immediately at the start of the fine-tuning process, causing the model to "forget" how to follow general prompts.
Methodology: The ProMoT Framework
To address format specialization, we propose ProMoT (PROmpt Tuning with MOdel Tuning), a two-stage framework designed to decouple format learning from semantic logic. By offloading rigid formatting requirements to a removable module, the model's core weights are preserved for deeper task reasoning.
- Stage 1: Prompt Tuning: The pre-trained model is frozen, and only a "soft prompt"—a set of tunable parameters prepended to the input—is trained. This prompt is designed to "absorb" the task's specific formatting requirements.
- Stage 2: Model Tuning: The soft prompt is frozen, and the main model parameters are unfrozen and tuned. Since the format is already handled by the prompt, the model's internal weights are forced to focus on the underlying logic and semantics of the task.
At inference, the soft prompt can be kept for specialized tasks or removed to reveal a core model that retains its general intelligence.
Key Results and Insights
Experiments using models like mT5 XXL (13B) and PaLM 8b demonstrate that ProMoT effectively preserves general intelligence while maintaining high performance on specialized tasks.
As shown in the experimental results, standard fine-tuning on the RTE task causes performance on general tasks like TriviaQA to drop from 17.58% to nearly zero (0.15%). In contrast, ProMoT maintains the model's original performance levels (17.64%). Furthermore, ProMoT can lead to semantic synergy. For instance, fine-tuning on English-French translation using ProMoT improved performance on other language pairs (e.g., English-German) by over 15 BLEU points compared to standard fine-tuning, which essentially caused the model to "forget" other languages entirely.
Conclusion
ProMoT provides a simple yet effective method for specializing LLMs without the cost of general-purpose degradation. By separating "how to talk" (format) from "what to say" (logic), ProMoT ensures that fine-tuned models remain versatile and capable across a broad range of unseen tasks. Future research will focus on testing ProMoT at the scale of frontier models exceeding 100 billion parameters.
Summary updated