Improving Text Auto-Completion with Next Phrase Prediction

Dong-Ho Lee, Zhiqiang Hu, Roy Ka-Wei Lee

Abstract

Language models such as GPT-2 have performed well on constructing syntactically sound sentences for text auto-completion task. However, such models often require considerable training effort to adapt to specific writing domains (e.g., medical). In this paper, we propose an intermediate training strategy to enhance pre-trained language models' performance in the text auto-completion task and fastly adapt them to specific domains. Our strategy includes a novel self-supervised training objective called Next Phrase Prediction (NPP), which encourages a language model to complete the partial query with enriched phrases and eventually improve the model's text auto-completion performance. Preliminary experiments have shown that our approach is able to outperform the baselines in auto-completion for email and academic writing domains.
BibTeX
@inproceedings{lee2021improving,
  author = {Dong-Ho Lee and Zhiqiang Hu and Roy Ka-Wei Lee},
  title = {{Improving Text Auto-Completion with Next Phrase Prediction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=l5gwfIkR7Wj},
}

Loading related content...

Figure