The Case for a Single Model that can Both Generate Continuations and Fill-in-the-Blank

Daphne Ippolito, Liam Dugan, Emily Reif, Ann Yuan, Andy Coenen, Chris Callison-Burch

Introduction

In the current landscape of AI-assisted writing, a functional divide exists between different types of generative tasks. Traditionally, if a writer needs an AI to finish a sentence, they utilize an auto-regressive "continuation" model like GPT. Conversely, if the task involves inserting a missing word or phrase into the middle of a paragraph, a specialized "infilling" model is required. This research, presented at NAACL 2022 by researchers from the University of Pennsylvania and Google Research, challenges this binary approach. The authors propose a "Swiss Army Knife" model: a single architecture capable of handling both continuation and fill-in-the-blank (FITB) tasks while offering fine-grained control over length and content.

Methodology

The researchers focused on the T5 (Text-to-Text Transfer Transformer) architecture. Unlike decoder-only models that primarily process information in one direction, T5’s encoder-decoder structure allows it to simultaneously consider context on both sides of a blank.

Overview of the Fill-in-the-Blank + Continuation Model showing various input prompts and their corresponding generated outputs.

To achieve a multi-functional model, the team leveraged T5's original "span corruption" pre-training—a process akin to a massive game of Mad Libs—and further finetuned it on a balanced mix of tasks: 50% infilling (blank in the middle) and 50% continuation (blank at the end). To enhance user control, they introduced conditioning signals or "tags":

  • Length Control: Directing the model to generate approximately $N$ words.
  • Goal Conditioning: Providing specific keywords or phrases that the model must incorporate into the generated text.

Evaluation and Datasets

The models were evaluated against a massive 137B parameter decoder-only baseline (similar to LaMDA) using few-shot prompting. Testing was conducted across three diverse datasets to ensure robustness:

  • C4 (Colossal Clean Crawled Corpus): General web text.
  • Reddit Writing Prompts (RWP): Creative narrative stories.
  • ROC Stories: Five-sentence stories focused on commonsense logical progression.

Results and Discussion

The study revealed that infilling is essentially a "superset" task. A model capable of infilling can perform continuation by simply placing the blank at the end of a sequence. However, standard continuation models struggle with infilling because they cannot effectively "look ahead" to ensure the inserted text fits the subsequent context.

Mean human ratings for Fill-in-the-blank and Continuation tasks, comparing Human, T5 variants, and LLM baselines.

Key findings include:

  • Efficiency over Scale: The 800M parameter T5 model outperformed the 137B parameter LLM on FITB tasks, suggesting that specialized architecture is more critical than raw parameter count for these specific writing aids.
  • Human Preference: In human evaluations, the finetuned T5 was rated nearly as high as the 137B LLM for continuation and was virtually indistinguishable from the pre-trained version in FITB quality.
  • Controllability: The finetuned model demonstrated high "obedience," hitting target length buckets 74% of the time and incorporating requested goal words in 79% of instances. While pre-trained T5 is naturally good at filling blanks, it tends to produce very short responses; finetuning was essential for following complex user instructions.

Conclusion

This research demonstrates that a single, moderately sized encoder-decoder model can serve as a versatile tool for various writing tasks. By unifying continuation and infilling into one model, developers can reduce resource overhead while providing writers with more flexible and controllable AI partners. Future work may expand these findings to technical domains like legal or medical writing and explore deeper integration into real-time collaborative interfaces.

Summary updated

Abstract

The task of inserting text into a specified position in a passage, known as fill in the blank (FitB), is useful for a variety of applications where writers interact with a natural language generation (NLG) system to craft text. While previous work has tackled this problem with models trained specifically to do the fill-in-the-blank task, a more useful model is one that can effectively perform _both_ FitB and continuation. In this work, we evaluate the feasibility of using a single model to do both tasks. We show that models pre-trained with a FitB-style objective are capable of both tasks, while models pre-trained for continuation are not. Finally, we show how FitB models can be easily finetuned to allow for fine-grained control over the length and word choice of the generation.
BibTeX
@inproceedings{ippolito2022case,
  author = {Daphne Ippolito and Liam Dugan and Emily Reif and Ann Yuan and Andy Coenen and Chris Callison-Burch},
  title = {{The Case for a Single Model that can Both Generate Continuations and Fill-in-the-Blank}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=oklOXVQdVj},
}
Figure