Abstract
Prompt tuning (PT) has emerged as a parameter-efficient alternative to full-model fine-tuning for adapting Large Language Models (LLMs) to downstream tasks. However, PT typically requires substantial labeled data, often underperforming in few-shot settings. This paper introduces a novel framework that leverages a generative LLM to synthesize task-specific training data. Central to this approach is DawGen (Distribution-aligned weighted Generator tuning), which ensures synthetic data aligns with the target task's distribution. By employing "gradient surgery" to manage the integration of real and synthetic data, the authors demonstrate that prompt tuning can surpass full-model fine-tuning performance while utilizing 7,500x fewer trainable parameters.
Introduction
As Large Language Models (LLMs) grow in scale, traditional full-model fine-tuning (FT) becomes increasingly prohibitive due to high computational costs. Prompt Tuning (PT) offers a solution by freezing the base model and training only a small set of continuous "soft prompt" vectors. Despite its efficiency, PT is notoriously "data-hungry." In few-shot scenarios—where only a handful of labeled examples are available—PT often fails to match the performance of FT.
Previous attempts to bridge this gap involved pre-training prompts on massive general-purpose datasets. However, these datasets frequently suffer from domain mismatch. This research explores a different path: using a powerful generative LLM to "dream up" a large-scale synthetic dataset based on a few real-world examples, thereby providing the necessary volume of data for effective prompt tuning.
Methodology: The DawGen Framework
The proposed framework consists of three primary stages designed to generate high-quality synthetic data and use it effectively for prompt tuning.
1. DawGen (Distribution-aligned weighted Generator tuning)
To ensure the generative LLM produces relevant and high-quality text, the authors introduce DawGen. This method tunes the generator using the available few-shot samples through two key mechanisms:
- Weighted Tuning: A "weight net" identifies tokens critical for class distinction (e.g., identifying sentiment-bearing words), focusing the generator on task-relevant features.
- Distribution Alignment: A regularization term forces the generator to produce sentences semantically similar to real examples within the same class while maintaining distance from other classes.
2. Synthetic Data Generation
Once tuned, the generator produces a large-scale synthetic dataset ($D_{syn}$). This dataset acts as a "knowledge bridge," transferring latent task knowledge from a source LLM to the target model.
3. Prompt Tuning with Gradient Surgery
Training on a mix of high-quality real data and potentially noisy synthetic data requires careful handling. The authors utilize Gradient Surgery to prevent synthetic data from introducing harmful noise. If the gradient from a synthetic batch opposes the gradient from a real batch, the conflicting component is subtracted. This ensures that the high-fidelity real data always dictates the primary direction of learning.
Experimental Setup
The framework was evaluated across seven sentence-pair classification tasks:
- Paraphrase Detection: MRPC and QQP.
- Natural Language Inference (NLI): MNLI, SNLI, QNLI, RTE, and SICK. These tasks require the model to determine if a premise entails, contradicts, or is neutral toward a hypothesis.
The researchers used T5-large and Flan-T5-large (770M parameters) as the target models. The generative source model was CTRL.
Results and Discussion
The experimental results highlight the efficacy of the DawGen framework in overcoming the limitations of few-shot prompt tuning.
Key Findings
- Efficiency and Performance: Prompt tuning with DawGen, using only 102K trainable parameters, outperformed full-model fine-tuning (770M parameters) by an average of 3.8% on T5-large.
- Significant Improvements: Compared to standard PT, the framework improved accuracy by approximately 18% for T5-large and 15% for Flan-T5-large.
- Overcoming Overfitting: In few-shot settings, soft prompts often overfit to "shortcut tokens." DawGen’s distribution alignment mitigates this by forcing the model to capture broader semantic meanings.
- Instruction Tuning Synergy: Models like Flan-T5, which are already instruction-tuned, showed a greater capacity to follow soft prompts and benefited more from synthetic data "warm-ups."
Conclusion and Future Work
This research demonstrates that synthetic data generation is a viable path for making prompt tuning competitive in few-shot regimes. By aligning the synthetic distribution with real-world data and using gradient surgery, the authors achieve state-of-the-art parameter efficiency.
However, challenges remain. Generating thousands of examples is computationally intensive, and simple accuracy metrics may not fully capture the quality of synthetic text. Future work will focus on developing more explainable quality metrics and exploring faster inference techniques to reduce the overhead of data generation.
Summary updated