Introduction
PALM (Pre-training an Autoencoding & Autoregressive Language Model) is a novel framework designed to bridge the gap between text comprehension and generation in "context-conditioned" tasks. Unlike traditional models that focus on reconstructing corrupted inputs, PALM aligns its pre-training objective with real-world applications such as generative question answering and abstractive summarization. By jointly training a bidirectional encoder for comprehension and an autoregressive decoder for generation, PALM achieves state-of-the-art results across multiple benchmarks.
The Challenge of Context-Conditioned Generation
Most modern language models, such as BART or MASS, utilize denoising autoencoding objectives where the model reconstructs a corrupted version of the input sentence. While effective for learning linguistic structure, this approach creates a mismatch with downstream tasks like conversational response or question answering. In these scenarios, the model must read a long context and generate new text that follows logically, rather than simply fixing a broken sentence. PALM addresses this by shifting the focus from input reconstruction to context-conditioned generation.
Methodology
PALM utilizes a standard Transformer encoder-decoder architecture with three primary innovations:
1. Joint Modeling and Two-Stage Training
The training process is divided into two stages to ensure the model is both a proficient reader and a coherent writer. In the first stage, the encoder is trained as a bidirectional autoencoder (similar to BERT) to reconstruct masked tokens within a context. In the second stage, the encoder and decoder are trained jointly. The encoder provides high-level representations of the source text, while the decoder learns to autoregressively generate subsequent text.
2. The 80/20 Context-Output Split
To mimic real-world generation tasks, PALM employs a specific data partitioning strategy. From a contiguous fragment of text (up to 500 tokens), the first 80% is provided to the encoder as "context," while the decoder is tasked with predicting the remaining 20% as "output." This forces the model to learn the logical flow of information from a source passage to a generated response.
3. Pre-trained Copy Mechanism
A key architectural feature of PALM is the integration of a pointer-generator network during the pre-training phase. This mechanism allows the model to dynamically decide whether to "copy" a specific entity or word from the source text or "generate" a word from its general vocabulary. By learning this during pre-training, PALM becomes significantly more adept at handling rare words and entities.
Evaluation and Results
PALM was pre-trained on English Wikipedia and BookCorpus and evaluated on several major benchmarks:
- MS MARCO (Question Answering): PALM achieved the top rank on the official leaderboard (Dec 2019) with a Rouge-L score of 0.498.
- CNN/DailyMail (Summarization): The PALM LARGE model set a new state-of-the-art with a Rouge-1 score of 44.30.
- Conversational Response: In low-data settings, PALM demonstrated a 40% improvement in perplexity over previous SOTA models like MASS.
Qualitative analysis further reveals PALM's ability to perform non-trivial inferences, such as estimating values based on contextual clues, which simpler denoising models often fail to capture.
Conclusion
By aligning pre-training objectives with the requirements of context-conditioned generation, PALM demonstrates that deep comprehension is the foundation of high-quality text generation. The integration of joint modeling and a pre-trained copy mechanism allows the model to excel in tasks requiring logical continuation and precise entity handling. Future work may explore scaling the model to larger datasets and extending its capabilities to multilingual contexts.
Summary updated