Abstract
Self-supervised pre-training, such as BERT, MASS, and BART, has emerged as a powerful technique for natural language understanding and generation. Existing pre-training techniques employ autoencoding and/or autoregressive objectives to train Transformer-based models by recovering original word tokens from corrupted text with some masked tokens. The training goals of existing techniques are often inconsistent with the goals of many language generation tasks, such as generative question answering and conversational response generation, for producing new text given context. This work presents PALM with a novel scheme that jointly pre-trains an autoencoding and autoregressive language model on a large unlabeled corpus, specifically designed for generating new text conditioned on context. The new scheme alleviates the mismatch introduced by the existing denoising scheme between pre-training and fine-tuning where generation is more than reconstructing original text. An extensive set of experiments show that PALM achieves new state-of-the-art results on a variety of language generation benchmarks covering generative question answering (Rank 1 on the official MARCO leaderboard), abstractive summarization on CNN/DailyMail as well as Gigaword, question generation on SQuAD, and conversational response generation on Cornell Movie Dialogues.
Introduction
In recent years, the "pre-train then fine-tune" paradigm (e.g., BERT, GPT, and BART) has dominated Natural Language Processing. However, a significant mismatch exists between how these models are pre-trained and how they are utilized for context-conditioned generation. Most generative models rely on a denoising objective: they take a corrupted sentence and attempt to reconstruct the original. While effective for general representation learning, this does not perfectly mirror tasks like Generative QA or Summarization. In these downstream tasks, the goal is not to "fix" the input, but to comprehend a long context and generate a new, subsequent piece of text. PALM (Pre-training an Autoencoding & Autoregressive Language Model) addresses this by shifting the pre-training objective from simple denoising to "comprehending and continuing."
Methodology
PALM utilizes a standard Transformer encoder-decoder architecture but introduces a unique two-stage pre-training scheme and a specific data representation designed to bridge the gap between comprehension and generation.
Two-Stage Pre-training
- Stage 1 (Encoder Only): The encoder is first trained as a bidirectional autoencoder, similar to BERT’s Masked Language Modeling, to ensure it can deeply comprehend text.
- Stage 2 (Joint Training): The encoder and decoder are jointly trained. To mimic downstream tasks, the model uses an 80/20 split: a contiguous text fragment of length $L$ is divided such that the first 80% is fed into the encoder as "context," and the decoder is tasked with generating the remaining 20% as the natural continuation.
Integrated Copy Mechanism
Crucially, PALM incorporates a pointer-generator network directly into the pre-training phase. This allows the model to learn when to "copy" specific entities or rare words from the context versus when to "generate" them from its internal vocabulary. By learning this mechanism on massive unlabeled data rather than just during fine-tuning, PALM develops a more robust logic for referencing context-specific entities.
Experimental Results
PALM was evaluated across several major benchmarks, demonstrating significant improvements over previous state-of-the-art (SOTA) models like MASS and BART:
- Generative QA (MARCO): PALM achieved Rank 1 on the official leaderboard (as of Dec 2019) with a Rouge-L score of 0.498, outperforming complex ensemble models.
- Abstractive Summarization: On the CNN/DailyMail dataset, PALM reached a Rouge-1 score of 44.30, surpassing both BART and PEGASUS.
- Conversational Response: In dialogue generation on the Cornell Movie Dialogues dataset, PALM showed a massive improvement in Perplexity (21.98) compared to MASS (23.52).
- Zero-Shot Inference: Qualitatively, the pre-trained model demonstrated a surprising ability to reason and infer facts from context without any fine-tuning, such as estimating auction prices or describing geographic locations.
Conclusion
PALM demonstrates that predicting the continuation of a text is a superior proxy for downstream generation tasks compared to simple reconstruction. By forcing the decoder to rely on the encoder's representations to predict the "future" of a text fragment, PALM creates a tighter coupling between comprehension and generation. Future work will focus on scaling PALM to even larger datasets and architectures to further push the boundaries of context-conditioned text generation.
Summary updated