PRINCE: Prefix-Masked Decoding for Knowledge Enhanced Sequence-to-Sequence Pre-Training

Song Xu, Haoran Li, Peng Yuan, Youzheng Wu, Xiaodong HeEMNLP 2022 · DOI

Abstract

PRINCE (PRefIx-masked decoding for kNowledge enhanCEd pre-training) is a novel pre-training paradigm for sequence-to-sequence (Seq2Seq) models designed to improve factual accuracy in data-to-text generation. By masking the prefix tokens of entities in the decoder, PRINCE forces the model to rely on encoder-side context rather than simple linguistic patterns. This approach achieves state-of-the-art performance on knowledge-intensive benchmarks like WebNLG and WikiBio.

Introduction

Modern language models typically rely on denoising autoencoder objectives for pre-training, where a model reconstructs a corrupted input. While effective for general fluency, models like BART and T5 often struggle with factual consistency in knowledge-heavy tasks. The authors identify a "decoder shortcut": when predicting multi-token entities (e.g., "New York City"), the decoder often relies on the ground-truth prefix ("New") to predict the subsequent token ("York"), essentially performing simple language modeling rather than retrieving factual knowledge from the input. PRINCE addresses this gap by introducing prefix-masking in the decoder to ensure the model internalizes underlying factual relationships.

Methodology: Prefix-Masked Decoding

The core innovation of PRINCE is the modification of the decoder's training objective. In standard Seq2Seq pre-training, such as MASS, the decoder is provided with the previous ground-truth tokens to predict the next token. PRINCE disrupts this by replacing the preceding tokens of an entity with a [MASK] symbol.

Comparison between MASS and PRINCE pre-training paradigms.

As illustrated in Figure 1, if the target sequence is "Tom Cruise was born in New York City," and the model is predicting "York," a standard decoder sees "New." In PRINCE, "New" is masked. This forces the model to leverage the encoder's representation of the source data (e.g., structured RDF triples) to generate the correct entity. The architecture is based on the BART-large framework, featuring a 12-layer encoder and a 12-layer decoder.

Experimental Setup

The model was pre-trained on a 14GB subset of English Wikipedia, with entities aligned to Wikidata to facilitate targeted masking. Evaluation was conducted on two primary data-to-text benchmarks:

  1. WebNLG (2.0): Generating natural language from RDF triples.
  2. WikiBio: Generating biography paragraphs from Wikipedia infoboxes.

Results and Analysis

PRINCE achieved new state-of-the-art results across both benchmarks. On WebNLG, it reached a BLEU score of 66.86, outperforming standard BART and specialized models like KGPT.

Performance comparison on the WebNLG dataset.

On the WikiBio dataset, PRINCE achieved a BLEU score of 49.03. Beyond automated metrics, human evaluation confirmed a significant improvement in faithfulness. PRINCE-generated text was found to be factually accurate 99% of the time, compared to 94% for BART and 87% for a standard Transformer without pre-training.

Key insights from the study include:

  • Entity-Centric Noising: Masking entities is significantly more effective than masking common tokens for knowledge-heavy tasks.
  • Warm-starting: Initializing PRINCE with a general-purpose model like BART preserves linguistic fluency while enhancing knowledge retrieval.

Conclusion and Future Work

PRINCE demonstrates that challenging the decoder during pre-training is essential for building knowledge-aware Seq2Seq models. While highly effective for data-to-text tasks, future research could explore the benefits of prefix-masking in general text-to-text tasks like summarization. Additionally, expanding the definition of "knowledge" beyond Wikidata entities to include lexical relations or domain-specific schemas remains a promising direction.

Summary updated

BibTeX
@inproceedings{xu2022prince,
  author = {Song Xu and Haoran Li and Peng Yuan and Youzheng Wu and Xiaodong He},
  title = {{PRINCE: Prefix-Masked Decoding for Knowledge Enhanced Sequence-to-Sequence Pre-Training}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.171},
}
Figure