POINTER: Constrained Text Generation via Insertion-based Generative Pre-training

Yizhe Zhang, Guoyin Wang, Chunyuan Li, Zhe Gan, Chris Brockett, Bill DolanarXiv 2020 · OpenReview

TL;DR

Generating text with specific keywords is tricky for large language models. POINTER is a new method that progressively inserts words into a text, building it up from high-level concepts to fine details. This approach achieves state-of-the-art results on constrained text generation tasks and offers significantly faster inference times.

What Problem Does This Paper Address?

Large language models like BERT and GPT-2 are fantastic at generating free-form text, but they struggle when you need them to include specific words or phrases in a particular order (hard-constrained text generation). This is a common requirement in real-world applications, such as summarizing meeting notes with key phrases, rephrasing search queries, or generating stories with specific keywords. Existing methods either fail to guarantee the inclusion of all keywords or are computationally very expensive.

Methods

The core innovation is POINTER (PrOgressive INsertion-based TransformER). Instead of generating text word-by-word from left to right (autoregressive), POINTER works by inserting new tokens into existing text. This process is done in stages, creating a coarse-to-fine hierarchy.

Here's how it works:

  • Progressive Insertion: The model starts with the given keywords and progressively inserts new tokens between existing ones. This is done in a parallel, non-autoregressive manner.
  • Coarse-to-Fine Hierarchy: In early stages, the model inserts "important" words (like nouns and verbs) that form the skeleton of the sentence. Later stages fill in less important words (like articles and prepositions). This makes the generation process more intuitive and interpretable.
  • Pre-training: The authors pre-train POINTER on a massive Wikipedia dataset using a novel insertion-based objective. This allows the model to learn general language generation skills before being fine-tuned for specific constrained tasks.
  • DP-based Data Pair Construction: To train the model effectively, they create training data by "masking" tokens from a complete sentence to simulate the reverse of the insertion process. They use dynamic programming to intelligently select which tokens to mask, prioritizing important ones.
  • Inner-Layer Beam Search (ILBS): To address the potential for generating repetitive or inconsistent text in non-autoregressive models, they introduce a modified beam search that operates within each "layer" or stage of insertion, improving generation quality.

Datasets

  • Pre-training: A 12GB English Wikipedia dataset.
  • Fine-tuning and Evaluation:
    • News dataset (EMNLP2017 WMT News dataset)
    • Yelp English review dataset

Key Insights and Ideas

  • Hierarchy is Key for Constraints: The idea of building text from high-level structure (important words) to fine details (auxiliary words) is crucial for successfully incorporating hard constraints. It mirrors how humans might construct a sentence with specific requirements.
  • Insertion is More Natural for Constraints: Unlike left-to-right generation, inserting tokens allows the model to "grow" the sentence around the required keywords, making it easier to ensure they are present and in the correct order.
  • Pre-training on Wikipedia Pays Off: Leveraging a large, general-domain dataset for pre-training significantly boosts performance when fine-tuning on specific downstream tasks, even for constrained generation.
  • Speedup is Empirically Significant: While theoretically not guaranteed to be O(log N), the progressive, parallel nature of POINTER leads to empirically logarithmic inference time complexity in practice, a major advantage over many existing methods.

Key Results

  • State-of-the-Art Performance: POINTER achieved state-of-the-art results on both News and Yelp datasets for hard-constrained text generation, outperforming strong baselines like CGMH and NMSTG across various metrics (NIST, BLEU, METEOR).
  • Improved Efficiency: The non-autoregressive decoding leads to significantly faster inference times. The paper reports that 98% of generations finish within 4 stages, and inference is empirically much faster than traditional methods.
  • Human Evaluation: Human judges preferred POINTER's outputs for semantic consistency, fluency, and informativeness over baseline methods.

Limitations and Future Work

  • Heuristic Importance Scoring: The method for scoring token importance (TF-IDF, POS, YAKE) is heuristic. The authors suggest that an unbiased, oracle assessment of importance would be beneficial.
  • Sentence Structure: The authors plan to explore leveraging sentence structure, such as constituency parsing, to further enhance the progressive hierarchy.
  • Flexibility: They also suggest extending the model to handle inflected/variant forms of keywords and arbitrary ordering of constraints.
  • ILBS Speed: While ILBS improves quality, it is slower than greedy decoding. The paper notes that the large model with ILBS is time-consuming.

Summary updated

Abstract

Large-scale pre-trained language models, such as BERT and GPT-2, have achieved excellent performance in language representation learning and free-form text generation. However, these models cannot be directly employed to generate text under specified lexical constraints. To address this challenge, we present POINTER (PrOgressive INsertion-based TransformER), a simple yet novel insertion-based approach for hard-constrained text generation. The proposed method operates by progressively inserting new tokens between existing tokens in a parallel manner. This procedure is recursively applied until a sequence is completed. The resulting coarse-to-fine hierarchy makes the generation process intuitive and interpretable. We pre-train our model with the proposed progressive insertion-based objective on a 12GB Wikipedia dataset, and fine-tune it on downstream hard-constrained generation tasks. Non-autoregressive decoding yields an empirically logarithmic time complexity during inference time. Experimental results on both News and Yelp datasets demonstrate that POINTER achieves state-of-the-art performance on constrained text generation. We released the pre-trained models and the source code to facilitate future research (https://github.com/dreasysnail/POINTER).
BibTeX
@misc{zhang2020pointer,
  author = {Yizhe Zhang and Guoyin Wang and Chunyuan Li and Zhe Gan and Chris Brockett and Bill Dolan},
  title = {{POINTER: Constrained Text Generation via Insertion-based Generative Pre-training}},
  year = {2020},
  eprint = {2005.00558},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2005.00558},
}
Figure