Improving Text Auto-Completion with Next Phrase Prediction

Dong-Ho Lee, Zhiqiang Hu, Roy Ka-Wei Lee

Abstract

Text auto-completion systems often struggle with domain-specific terminology. We present Next Phrase Prediction (NPP), an intermediate training strategy that teaches models to predict meaningful syntactic phrases (NP, VP, PP) rather than individual words. By bridging the gap between general pre-training and domain-specific fine-tuning, NPP+T5 achieves state-of-the-art performance in specialized domains like academic writing and professional emails, outperforming larger models like GPT-2.

Introduction

While large language models (LLMs) like GPT-2 excel at generating grammatically correct sentences, they frequently struggle with domain adaptation. In technical or professional contexts, standard LLMs often suggest generic continuations rather than the specific terminology required. Current solutions typically rely on massive fine-tuning on domain-specific data, which is computationally expensive and data-intensive.

The authors identify a critical research gap: making pre-trained models "smarter" about specialized language structure without astronomical amounts of new data. They propose that the solution lies in understanding phrases—the semantic building blocks between individual words and full sentences.

Methods: Next Phrase Prediction (NPP)

The proposed approach introduces an intermediate training step using a novel self-supervised objective called Next Phrase Prediction (NPP). The workflow follows a three-stage process: starting with a standard T5 transformer, performing intermediate training on a general corpus (Wikipedia) using the NPP objective, and finally fine-tuning on the target domain.

The NPP objective involves two primary steps:

  1. Phrase Extraction: Using constituency parsing, the system identifies qualitative phrases, specifically Noun Phrases (NP), Verb Phrases (VP), and Prepositional Phrases (PP).
  2. Generative Question Answering (QA): The model is presented with a partial sentence and candidate phrases. It must learn to select the correct phrase to complete the sentence.
The NPP workflow: from constituency parsing to generative phrase selection.

Experimental Setup

The model was evaluated on two distinct target domains:

  • Email: The Enron email corpus, representing professional communication.
  • Academic Writing: Abstracts from ArnetMiner (Computer Science), representing a high-expertise technical domain.

The authors noted a "Difficulty Gap," where academic writing proved significantly harder for models to master than emails, with Bi-LSTM baselines showing nearly double the perplexity on academic text.

Results

The proposed NPP+T5 model was compared against GPT-2, T5, and T5 trained with Next Sentence Prediction (NSP). NPP+T5 consistently outperformed all baselines across multiple metrics, including BLEU-4, METEOR, CIDEr, and SPICE.

Performance comparison of NPP+T5 against baselines across Email and Academic Writing domains.

Notably, NPP+T5 outperformed GPT-2 despite having significantly fewer parameters (220M vs 774M). Qualitatively, the NPP-trained model provided more relevant technical suggestions. For instance, given a prompt about "Kalman filters," the base T5 suggested generic terms like "et al.," while the NPP model correctly suggested "filter is established."

Conclusion

Next Phrase Prediction serves as an effective "warm-up" task, teaching models the mechanics of completion before they learn domain-specific vocabulary. While currently tested on Email and Computer Science domains, future work will explore more diverse styles such as medical or legal writing. The authors also plan to scale the intermediate training beyond the current 1.2 billion tokens to further enhance performance.

Summary updated

Abstract

Language models such as GPT-2 have performed well on constructing syntactically sound sentences for text auto-completion task. However, such models often require considerable training effort to adapt to specific writing domains (e.g., medical). In this paper, we propose an intermediate training strategy to enhance pre-trained language models' performance in the text auto-completion task and fastly adapt them to specific domains. Our strategy includes a novel self-supervised training objective called Next Phrase Prediction (NPP), which encourages a language model to complete the partial query with enriched phrases and eventually improve the model's text auto-completion performance. Preliminary experiments have shown that our approach is able to outperform the baselines in auto-completion for email and academic writing domains.
BibTeX
@inproceedings{lee2021improving,
  author = {Dong-Ho Lee and Zhiqiang Hu and Roy Ka-Wei Lee},
  title = {{Improving Text Auto-Completion with Next Phrase Prediction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=l5gwfIkR7Wj},
}
Figure