Abstract
Text auto-completion systems often struggle with domain-specific terminology. We present Next Phrase Prediction (NPP), an intermediate training strategy that teaches models to predict meaningful syntactic phrases (NP, VP, PP) rather than individual words. By bridging the gap between general pre-training and domain-specific fine-tuning, NPP+T5 achieves state-of-the-art performance in specialized domains like academic writing and professional emails, outperforming larger models like GPT-2.
Introduction
While large language models (LLMs) like GPT-2 excel at generating grammatically correct sentences, they frequently struggle with domain adaptation. In technical or professional contexts, standard LLMs often suggest generic continuations rather than the specific terminology required. Current solutions typically rely on massive fine-tuning on domain-specific data, which is computationally expensive and data-intensive.
The authors identify a critical research gap: making pre-trained models "smarter" about specialized language structure without astronomical amounts of new data. They propose that the solution lies in understanding phrases—the semantic building blocks between individual words and full sentences.
Methods: Next Phrase Prediction (NPP)
The proposed approach introduces an intermediate training step using a novel self-supervised objective called Next Phrase Prediction (NPP). The workflow follows a three-stage process: starting with a standard T5 transformer, performing intermediate training on a general corpus (Wikipedia) using the NPP objective, and finally fine-tuning on the target domain.
The NPP objective involves two primary steps:
- Phrase Extraction: Using constituency parsing, the system identifies qualitative phrases, specifically Noun Phrases (NP), Verb Phrases (VP), and Prepositional Phrases (PP).
- Generative Question Answering (QA): The model is presented with a partial sentence and candidate phrases. It must learn to select the correct phrase to complete the sentence.
Experimental Setup
The model was evaluated on two distinct target domains:
- Email: The Enron email corpus, representing professional communication.
- Academic Writing: Abstracts from ArnetMiner (Computer Science), representing a high-expertise technical domain.
The authors noted a "Difficulty Gap," where academic writing proved significantly harder for models to master than emails, with Bi-LSTM baselines showing nearly double the perplexity on academic text.
Results
The proposed NPP+T5 model was compared against GPT-2, T5, and T5 trained with Next Sentence Prediction (NSP). NPP+T5 consistently outperformed all baselines across multiple metrics, including BLEU-4, METEOR, CIDEr, and SPICE.
Notably, NPP+T5 outperformed GPT-2 despite having significantly fewer parameters (220M vs 774M). Qualitatively, the NPP-trained model provided more relevant technical suggestions. For instance, given a prompt about "Kalman filters," the base T5 suggested generic terms like "et al.," while the NPP model correctly suggested "filter is established."
Conclusion
Next Phrase Prediction serves as an effective "warm-up" task, teaching models the mechanics of completion before they learn domain-specific vocabulary. While currently tested on Email and Computer Science domains, future work will explore more diverse styles such as medical or legal writing. The authors also plan to scale the intermediate training beyond the current 1.2 billion tokens to further enhance performance.
Summary updated