Abstract
Pretrained Language Models (PLMs) have significantly advanced natural language processing, yet they remain sensitive to data scarcity during fine-tuning. While prompt-based learning offers a lightweight alternative, existing methods often struggle with task-specific rigidity and a lack of cross-task transferability. This paper introduces Prompt Transfer for Text Generation (PTG), a novel framework designed to transfer "soft prompts" from source tasks to new target domains. By utilizing an adaptive attention mechanism that integrates both task-level and instance-specific information, PTG achieves superior performance in data-scarce scenarios while maintaining a frozen model backbone.
Introduction
Fine-tuning massive models like BART, T5, and GPT-3 typically requires substantial labeled datasets, which are often unavailable for niche domains or emerging tasks. While "prompting"—the practice of prepending task-specific instructions or continuous vectors to inputs—has emerged as a parameter-efficient alternative, it faces two primary limitations. First, prompts are usually highly specialized, making them difficult to reuse across different generation types. Second, most prompting methods utilize "static" prompts that do not adapt to the nuances of individual data instances.
PTG addresses these challenges by creating a transferable prompting architecture. It treats prompts as modular units of "generation knowledge" that can be retrieved and combined dynamically based on the context of the target task and the specific input provided.
The PTG Framework
The PTG architecture is built upon four primary components designed to facilitate efficient knowledge transfer:
- Source Prompt Pool: The system begins by training independent "soft prompts" (continuous vectors) on a diverse array of source tasks, such as summarization and dialogue generation.
- Clustering and Memory Network: To manage these prompts effectively, PTG employs spectral clustering to group similar source prompts. These are organized within a Multi-Key Memory Network, where each prompt is indexed by both a "cluster key" and a "prompt key."
- Adaptive Attention Mechanism: This core innovation allows the model to retrieve and combine source prompts using two distinct queries:
- Task-level Query: A learnable vector that identifies which source tasks are broadly relevant to the target task.
- Instance-level Query: A query derived from the specific input text (using a frozen BERT encoder) to identify source prompts that match the immediate context.
- Frozen Backbone: Throughout the transfer process, the underlying PLM (e.g., BART-Large) remains frozen. Only the attention mechanism and prompt keys are updated, ensuring the model remains lightweight and computationally efficient.
Experimental Setup
The researchers evaluated PTG across 14 public datasets, categorized into three functional groups to test the breadth of the transfer capabilities:
- Compression: Summarization (CNN/Daily Mail, XSum, MSNews, Multi-News, NEWSROOM) and question generation (SQuAD).
- Transduction: Style transfer (Wiki Neutrality) and paraphrase (Quora).
- Creation: Dialogue (PersonaChat, TopicalChat, DailyDialog, DSTC7-AVSD, MultiWOZ) and story generation (WritingPrompts).
Key Insights and Results
Few-Shot Dominance
PTG demonstrates significant advantages in low-data regimes. In experiments ranging from 50 to 500 training instances, PTG consistently outperformed standard fine-tuned models and existing prompt-based baselines like PrefixTuning and SPOT. This suggests that the transferred knowledge from source prompts effectively compensates for the lack of target-specific labels.
The Importance of Instance-Level Adaptation
A critical finding of the study is that task-wide prompts are often insufficient. By incorporating an instance-level query, PTG can pivot between different source prompts based on the input content. For example, a dialogue task might retrieve "storytelling" prompts for narrative inputs and "summarization" prompts for factual exchanges. Ablation studies confirmed that removing either the instance-level query or the clustering step resulted in significant performance degradation.
Implicit Task Mapping
Clustering analysis revealed that PTG naturally organizes tasks in alignment with human intuition. The model successfully clustered "creative" tasks (story generation and dialogue) together, while keeping "compression" tasks (news summarization) in a separate functional group. This indicates that soft prompts capture meaningful structural similarities between different NLP tasks.
Conclusion and Future Work
PTG represents a significant step toward making large-scale text generation models more accessible for data-scarce applications. By treating prompts as a transferable knowledge base, the framework achieves competitive results with minimal trainable parameters.
Future research directions include expanding the diversity of the source prompt pool and developing an open-resource library for automatic prompt reuse. Additionally, the authors acknowledge the ethical necessity of monitoring these models, as the high quality of generated text increases the difficulty of distinguishing machine-generated content from human-written text.
Summary updated