MVP: Multi-task Supervised Pre-training for Natural Language Generation

Tianyi Tang, Junyi Li, Xin Zhao, Ji-Rong Wen

Introduction

The landscape of Natural Language Generation (NLG) has long been dominated by models like BART and T5, which rely primarily on unsupervised pre-training. These models learn by "denoising" text—predicting missing words or reordering sentences from massive, unlabeled web crawls. While effective, this approach introduces noise and creates a significant "objective gap" between pre-training (fixing broken text) and downstream tasks like summarization or dialogue.

The MVP (Multi-task superVised Pre-training) model introduces a paradigm shift by leveraging supervised signals. While supervised pre-training (instruction tuning) has revolutionized Natural Language Understanding (NLU) with models like FLAN and T0, MVP brings this power to the generation side, arguing that supervised signals are more natural for learning specific input-to-output transformations.

Methodology

The MVP framework utilizes a two-stage approach to build a versatile and specialized generation model.

Overview of the two-stage MVP pre-training process.

Stage 1: Backbone Pre-training

The model uses a standard Transformer encoder-decoder architecture, initially seeded with BART-large weights. This backbone is pre-trained on a diverse mixture of labeled tasks. To unify these disparate datasets, human-written instructions (e.g., "Summarize:") are used to create a consistent text-to-text format.

Stage 2: Task-specific Prompt Pre-training

To prevent "task interference"—where different tasks might neutralize specialized knowledge—the researchers introduced soft prompts. These are trainable continuous vectors prepended to attention modules. During this stage, the main backbone is frozen, and seven distinct groups of prompts are trained for specific task categories like "Dialogue" or "Data-to-text." This allows the model to "plug in" specialized modules while maintaining a general understanding.

The MVPCorpus

Central to MVP's success is the MVPCorpus, a massive collection of labeled NLG data. It consists of 77 datasets covering 11 diverse tasks, including data-to-text, open-ended dialogue, story generation, and summarization. The model's generalization was tested on 23 "seen" datasets and 4 "unseen" tasks, such as style transfer and commonsense generation.

Key Results

MVP demonstrates significant performance gains over traditional unsupervised and instruction-tuned models. It achieved state-of-the-art (SOTA) results on 13 out of 17 benchmark datasets.

Performance comparison of MVP against BART, Flan-T5, and other baselines across multiple benchmarks.

In head-to-head comparisons, MVP outperformed its base model, BART, by 9.3% and the instruction-tuned Flan-T5 by 5.8% in full-tuning settings. Notably, MVP's zero-shot performance surpassed the much larger T0-11B model, highlighting the efficiency of high-quality supervised pre-training. Furthermore, in parameter-efficient "lightweight" tuning, MVP with pre-trained prompts (MVP+S) outperformed standard prefix-tuning on BART by 4.3%.

Conclusion

MVP proves that supervised pre-training is inherently better suited for NLG than unsupervised denoising. By providing task-related instructions early and using soft prompts to store specialized knowledge, MVP achieves superior performance with fewer parameters than many larger models. Future work will explore scaling this approach to 10B+ parameter models and expanding support to multilingual contexts.

Summary updated

Abstract

Pre-trained language models (PLMs) have achieved remarkable success in natural language generation (NLG) tasks. Up to now, most NLG-oriented PLMs are pre-trained in an unsupervised manner using the large-scale general corpus. In the meanwhile, an increasing number of models pre-trained with labeled data (i.e., “supervised pre-training”) showcase superior performance compared to unsupervised pre-trained models. Motivated by the success of supervised pre-training, we propose Multi-task superVised Pre-training (MVP) for natural language generation. We collect a large-scale natural language generation corpus, MVPCorpus, from $77$ datasets over $11$ diverse NLG tasks. Then we unify these examples into a general text-to-text format to pre-train the text generation model MVP in a supervised manner. For each task, we further pre-train specific soft prompts to stimulate the model’s capacity to perform a specific task. Extensive experiments have demonstrated the effectiveness and generalizability of our MVP model in a number of NLG tasks, which achieves state-of-the-art performance on $13$ out of $17$ datasets.
BibTeX
@inproceedings{tang2023mvp,
  author = {Tianyi Tang and Junyi Li and Xin Zhao and Ji-Rong Wen},
  title = {{MVP: Multi-task Supervised Pre-training for Natural Language Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.558},
}
Figure