PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation

Bin Bi, Chenliang Li, Chen Wu, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, Luo Si

Introduction

PALM (Pre-training an Autoencoding & Autoregressive Language Model) is a novel framework designed to bridge the gap between text comprehension and generation in "context-conditioned" tasks. Unlike traditional models that focus on reconstructing corrupted inputs, PALM aligns its pre-training objective with real-world applications such as generative question answering and abstractive summarization. By jointly training a bidirectional encoder for comprehension and an autoregressive decoder for generation, PALM achieves state-of-the-art results across multiple benchmarks.

The Challenge of Context-Conditioned Generation

Most modern language models, such as BART or MASS, utilize denoising autoencoding objectives where the model reconstructs a corrupted version of the input sentence. While effective for learning linguistic structure, this approach creates a mismatch with downstream tasks like conversational response or question answering. In these scenarios, the model must read a long context and generate new text that follows logically, rather than simply fixing a broken sentence. PALM addresses this by shifting the focus from input reconstruction to context-conditioned generation.

Methodology

PALM utilizes a standard Transformer encoder-decoder architecture with three primary innovations:

1. Joint Modeling and Two-Stage Training

The training process is divided into two stages to ensure the model is both a proficient reader and a coherent writer. In the first stage, the encoder is trained as a bidirectional autoencoder (similar to BERT) to reconstruct masked tokens within a context. In the second stage, the encoder and decoder are trained jointly. The encoder provides high-level representations of the source text, while the decoder learns to autoregressively generate subsequent text.

PALM's joint pre-training objective: autoencoding for comprehension and autoregression for generation.

2. The 80/20 Context-Output Split

To mimic real-world generation tasks, PALM employs a specific data partitioning strategy. From a contiguous fragment of text (up to 500 tokens), the first 80% is provided to the encoder as "context," while the decoder is tasked with predicting the remaining 20% as "output." This forces the model to learn the logical flow of information from a source passage to a generated response.

3. Pre-trained Copy Mechanism

A key architectural feature of PALM is the integration of a pointer-generator network during the pre-training phase. This mechanism allows the model to dynamically decide whether to "copy" a specific entity or word from the source text or "generate" a word from its general vocabulary. By learning this during pre-training, PALM becomes significantly more adept at handling rare words and entities.

The pointer-generator mechanism integrated into PALM's architecture allows for dynamic copying from the source context.

Evaluation and Results

PALM was pre-trained on English Wikipedia and BookCorpus and evaluated on several major benchmarks:

  • MS MARCO (Question Answering): PALM achieved the top rank on the official leaderboard (Dec 2019) with a Rouge-L score of 0.498.
  • CNN/DailyMail (Summarization): The PALM LARGE model set a new state-of-the-art with a Rouge-1 score of 44.30.
  • Conversational Response: In low-data settings, PALM demonstrated a 40% improvement in perplexity over previous SOTA models like MASS.

Qualitative analysis further reveals PALM's ability to perform non-trivial inferences, such as estimating values based on contextual clues, which simpler denoising models often fail to capture.

Conclusion

By aligning pre-training objectives with the requirements of context-conditioned generation, PALM demonstrates that deep comprehension is the foundation of high-quality text generation. The integration of joint modeling and a pre-trained copy mechanism allows the model to excel in tasks requiring logical continuation and precise entity handling. Future work may explore scaling the model to larger datasets and extending its capabilities to multilingual contexts.

Summary updated

Abstract

Self-supervised pre-training, such as BERT, MASS and BART, has emerged as a powerful technique for natural language understanding and generation. Existing pre-training techniques employ autoencoding and/or autoregressive objectives to train Transformer-based models by recovering original word tokens from corrupted text with some masked tokens. The training goals of existing techniques are often inconsistent with the goals of many language generation tasks, such as generative question answering and conversational response generation, for producing new text given context. This work presents PALM with a novel scheme that jointly pre-trains an autoencoding and autoregressive language model on a large unlabeled corpus, specifically designed for generating new text conditioned on context. The new scheme alleviates the mismatch introduced by the existing denoising scheme between pre-training and fine-tuning where generation is more than reconstructing original text. An extensive set of experiments show that PALM achieves new state-of-the-art results on a variety of language generation benchmarks covering generative question answering (Rank 1 on the official MARCO leaderboard), abstractive summarization on CNN/DailyMail as well as Gigaword, question generation on SQuAD, and conversational response generation on Cornell Movie Dialogues.
BibTeX
@inproceedings{bi2020palm,
  author = {Bin Bi and Chenliang Li and Chen Wu and Ming Yan and Wei Wang and Songfang Huang and Fei Huang and Luo Si},
  title = {{PALM: Pre-training an Autoencoding\&Autoregressive Language Model for Context-conditioned Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=iz7cYrN85rq},
}
Figure