Abstract
Traditional pre-training for natural language generation (NLG) typically utilizes unsupervised objectives, such as "fill-in-the-blank" tasks on raw text. However, this approach often suffers from a "discrepancy gap" between pre-training and the specific input-to-output transformations required for downstream tasks like summarization or dialogue. This paper introduces MVP, a model pre-trained using a massive, multi-task supervised approach. By leveraging the MVPCorpus—a curated collection of 77 labeled datasets—and a two-stage training process involving task-specific soft prompts, MVP significantly outperforms standard models like BART and Flan-T5. Our findings suggest that supervised pre-training provides a clearer signal for generative tasks, leading to superior performance and robust zero-shot generalization.
Introduction
Most modern language models follow a "pre-train then fine-tune" paradigm. While unsupervised pre-training on massive corpora has proven powerful, it is often sample-inefficient and fails to capture the specific instructional nature of many NLG tasks. The model spends significant compute learning general language statistics rather than the structured transformations required for practical applications.
MVP addresses this by shifting the pre-training phase from unsupervised learning to supervised multi-task learning. By training on labeled data from the outset, the model learns to follow instructions and perform specific generative tasks more effectively than models exposed only to raw text.
Methodology
The MVP framework consists of two primary stages: massive supervised pre-training and task-specific prompt tuning.
Stage 1: Massive Supervised Pre-training
The authors developed the MVPCorpus, a unified collection of labeled NLG data. Every dataset within the corpus was converted into a "text-to-text" format using human-written natural language instructions (e.g., "Summarize: [Document]"). The base model, a 406M parameter Transformer architecture (equivalent in size to BART-Large), was trained on this diverse mixture of tasks.
Stage 2: Task-Specific Soft Prompts
To mitigate task interference—where learning one task might degrade performance on another—the researchers introduced soft prompts. These are continuous, trainable vectors prepended to the model's layers. During this stage, the main MVP backbone is frozen, and these prompts are trained on specific task groups (e.g., all summarization data) to store specialized knowledge without altering the general capabilities of the core model.
Experimental Setup
The MVPCorpus comprises 77 datasets covering 11 distinct NLG tasks, including summarization, dialogue, data-to-text, question answering, and story generation. Evaluation was conducted on 23 "seen" datasets (tasks included in pre-training) and 4 "unseen" tasks (e.g., text simplification and style transfer) to assess the model's generalization capabilities.
Results and Discussion
MVP demonstrates substantial improvements over existing benchmarks. On average, MVP outperformed its unsupervised counterpart, BART, by 9.3% and the instruction-tuned Flan-T5 by 5.8% across 17 datasets. The model achieved state-of-the-art (SOTA) results on 13 out of 17 tested datasets.
Key Insights
- Supervised Signal: Labeled data provides a clearer objective for learning input-output mappings, making the model more robust than those trained on raw text alone.
- Instruction Following: By incorporating instructions during pre-training, MVP learns to follow directions as a fundamental capability rather than a post-hoc adjustment.
- Efficiency and Scale: Despite its relatively modest size (406M parameters), MVP outperformed the much larger 11-billion parameter T0 model in several zero-shot scenarios, highlighting the importance of training strategy over raw parameter count.
- Human Preference: Blind human evaluations confirmed that MVP’s outputs are preferred over BART’s in summarization, dialogue, and story generation.
Conclusion and Future Work
MVP represents a significant shift toward supervised pre-training for NLG. While currently focused on English and limited to a 406M parameter scale, the success of this approach opens several avenues for future research. Extending MVP to multilingual contexts and scaling the supervised pre-training approach to 10B+ parameter models could further enhance its generative capabilities and cross-task synergy.
Reference: Tang, T., Li, J., Zhao, W. X., & Wen, J. R. (2023). MVP: Multi-task Supervised Pre-training for Natural Language Generation. Findings of the Association for Computational Linguistics: ACL 2023.
Summary updated