TL;DR
Fine-tuning large language models (LLMs) on specific tasks often destroys their general "in-context learning" abilities, a phenomenon the authors attribute to format specialization. To fix this, they introduce ProMoT (Prompt Tuning with MOdel Tuning), a two-stage framework that offloads format learning to a soft prompt before updating the model's weights, preserving versatility without sacrificing task-specific accuracy.
What Problem Does This Paper Address?
LLMs are prized for their ability to solve diverse tasks via few-shot prompting (in-context learning). However, to squeeze out maximum performance for a specific application, researchers often fine-tune them on specialized datasets.
The catch? Standard fine-tuning usually causes the model to "forget" its general-purpose capabilities. For example, if you fine-tune a model on a binary classification task like Natural Language Inference (NLI), it quickly learns to output only "True" or "False." If you then ask that same model to summarize a paragraph, it might stubbornly respond with "True" because it has over-specialized to the NLI format.
The authors identify that this loss of generalization happens incredibly fast—often within the first few hundred steps of training—and is primarily caused by the model overfitting to the format of the data rather than its semantic content.
Methods
The authors propose ProMoT (Prompt Tuning with MOdel Tuning), a simple two-stage strategy designed to decouple format learning from semantic learning:
- Stage 1: Prompt Tuning: The pretrained model is frozen. A small set of continuous, trainable parameters (a "soft prompt") is prepended to the input and trained on the target task. Because format learning happens first, the soft prompt "absorbs" the task's specific output style and structure.
- Stage 2: Model Tuning: The soft prompt learned in Stage 1 is now frozen, and the main model weights are unfrozen and fine-tuned. Since the format is already handled by the prompt, the model's internal weight updates can focus on the deeper semantic nuances of the task.
At inference time, if the incoming task matches the fine-tuned format, the prompt is kept. If the user wants to perform a different general task (like summarization), the prompt is removed, revealing the model's preserved general-purpose capabilities.
Datasets
The researchers used a variety of datasets to test both specialization and generalization:
- Fine-tuning Tasks: RTE (Recognizing Textual Entailment), WMT14 En-Fr (Translation), SNLI, and OpenbookQA.
- Evaluation Tasks (Unseen during tuning):
- NLI: CB and WiC.
- Closed-book QA: TriviaQA and WebQuestions.
- Translation: WMT16 En-Ro and En-De.
- Summarization: XSum and WikiLingua.
Key Insights and Ideas
- Format Specialization is the Culprit: The authors found that at the very start of training, the model's gradients are highly aligned with the task format. In Figure 3, they show that the gradient for the actual task is almost identical to the gradient for a "randomized" version of the task where labels are shuffled (meaning the model is just learning the output labels, not the logic).
- Early Forgetting: General capabilities don't erode slowly; they collapse almost immediately. By step 300 of NLI fine-tuning, 90% of a model's outputs for a completely different QA task become "True" or "False."
- Semantic Synergy: Interestingly, ProMoT doesn't just preserve general skills; it can enhance them. Fine-tuning on NLI using ProMoT actually improved the model's performance on summarization tasks compared to the base pretrained model, likely because the model learned better "grounding" without being blinded by the NLI format.
Key Results
- Supervised Performance: ProMoT achieves comparable or even superior results on the target fine-tuning tasks compared to standard fine-tuning. For RTE, ProMoT reached 92.78% accuracy vs. 92.06% for standard tuning.
- Generalization: On a normalized average of 8 unseen tasks, standard fine-tuning on RTE caused a performance drop of -2.10%. In contrast, ProMoT achieved a gain of +2.58%.
- Cross-Task Boost: ProMoT trained on English-French translation significantly improved performance on other language pairs (like English-German), whereas standard fine-tuning caused those capabilities to drop to near zero.
Limitations and Future Work
- Model Scale: Most experiments were conducted on models with fewer than 15 billion parameters (mT5 XXL, PaLM 8b). The authors suggest testing ProMoT on even larger models.
- Format Absorption: While empirically successful, there is no theoretical guarantee that the soft prompt will absorb all format information in every possible scenario.
- Computational Cost: ProMoT requires a two-stage training process, which is slightly more complex to manage than a single-stage fine-tuning run.
Overall, ProMoT offers a practical path for engineers who need high-performance specialized models that don't lose their "magic" as general-purpose assistants.
Summary updated