Introduction
The landscape of Natural Language Generation (NLG) has long been dominated by models like BART and T5, which rely primarily on unsupervised pre-training. These models learn by "denoising" text—predicting missing words or reordering sentences from massive, unlabeled web crawls. While effective, this approach introduces noise and creates a significant "objective gap" between pre-training (fixing broken text) and downstream tasks like summarization or dialogue.
The MVP (Multi-task superVised Pre-training) model introduces a paradigm shift by leveraging supervised signals. While supervised pre-training (instruction tuning) has revolutionized Natural Language Understanding (NLU) with models like FLAN and T0, MVP brings this power to the generation side, arguing that supervised signals are more natural for learning specific input-to-output transformations.
Methodology
The MVP framework utilizes a two-stage approach to build a versatile and specialized generation model.
Stage 1: Backbone Pre-training
The model uses a standard Transformer encoder-decoder architecture, initially seeded with BART-large weights. This backbone is pre-trained on a diverse mixture of labeled tasks. To unify these disparate datasets, human-written instructions (e.g., "Summarize:") are used to create a consistent text-to-text format.
Stage 2: Task-specific Prompt Pre-training
To prevent "task interference"—where different tasks might neutralize specialized knowledge—the researchers introduced soft prompts. These are trainable continuous vectors prepended to attention modules. During this stage, the main backbone is frozen, and seven distinct groups of prompts are trained for specific task categories like "Dialogue" or "Data-to-text." This allows the model to "plug in" specialized modules while maintaining a general understanding.
The MVPCorpus
Central to MVP's success is the MVPCorpus, a massive collection of labeled NLG data. It consists of 77 datasets covering 11 diverse tasks, including data-to-text, open-ended dialogue, story generation, and summarization. The model's generalization was tested on 23 "seen" datasets and 4 "unseen" tasks, such as style transfer and commonsense generation.
Key Results
MVP demonstrates significant performance gains over traditional unsupervised and instruction-tuned models. It achieved state-of-the-art (SOTA) results on 13 out of 17 benchmark datasets.
In head-to-head comparisons, MVP outperformed its base model, BART, by 9.3% and the instruction-tuned Flan-T5 by 5.8% in full-tuning settings. Notably, MVP's zero-shot performance surpassed the much larger T0-11B model, highlighting the efficiency of high-quality supervised pre-training. Furthermore, in parameter-efficient "lightweight" tuning, MVP with pre-trained prompts (MVP+S) outperformed standard prefix-tuning on BART by 4.3%.
Conclusion
MVP proves that supervised pre-training is inherently better suited for NLG than unsupervised denoising. By providing task-related instructions early and using soft prompts to store specialized knowledge, MVP achieves superior performance with fewer parameters than many larger models. Future work will explore scaling this approach to 10B+ parameter models and expanding support to multilingual contexts.
Summary updated