The Case for a Single Model that can Both Generate Continuations and Fill in the Blank

Daphne Ippolito, Liam Dugan, Emily Reif, Ann Yuan, Andy Coenen, Chris Callison-Burch

Abstract

The task of inserting text into a specified position in a passage, known as fill in the blank (FitB), is useful for a variety of applications where writers interact with a natural language generation (NLG) system to craft text. While previous work has tackled this problem with models trained specifically to do the fill-in-the-blank task, a more useful model is one that can effectively perform both FitB and continuation. In this work, we evaluate the feasibility of using a single model to do both tasks. We show that models pre-trained with a FitB-style objective are capable of both tasks, while models pre-trained for continuation are not. Finally, we show how FitB models can be easily finetuned to allow for fine-grained control over the length and word choice of the generation.

Introduction

In the current landscape of large language models (LLMs), most architectures are designed as "completionists." These models, typically decoder-only transformers like the GPT family, excel at predicting the next token in a sequence. However, they struggle significantly when asked to perform "infilling" or Fill-in-the-Blank (FitB) tasks—inserting coherent text into a gap within an existing passage.

For human-AI collaborative writing, both capabilities are essential. A writer may want the AI to finish a paragraph (continuation) or suggest a sentence that bridges two existing ideas (infilling). Historically, these tasks were handled by separate models, which is computationally inefficient and complicates the user experience. This paper explores the potential for a single encoder-decoder model to master both tasks while providing fine-grained control over the output.

The Architectural Divide

The primary challenge in infilling for standard LLMs lies in their architecture. Decoder-only models are trained with a causal mask, meaning they only "see" the text that comes before the current word. When a gap exists in the middle of a paragraph, these models are blind to the "right-hand context"—the text following the gap. This leads to jarring transitions where the generated text fails to connect logically with what follows.

In contrast, encoder-decoder architectures like the Text-to-Text Transfer Transformer (T5) are naturally suited for this. T5 is pre-trained using a "span corruption" objective, where it learns to reconstruct missing spans of text. This pre-training effectively prepares the model to consider both preceding and succeeding context simultaneously.

Methodology

The researchers utilized T5 models (ranging from 800M to 3B parameters) and fine-tuned them on several objectives to bridge the gap between infilling and continuation:

  1. C4FILLBLANK: The model was trained to fill a gap located at a random position within a passage from the Colossal Clean Crawled Corpus (C4).
  2. C4FILLEND: The model was trained to treat the end of a passage as the blank, effectively teaching it to perform standard continuation.
  3. Goal Conditioning: To move beyond simple generation, the authors prepended instructions to the input. These included target length constraints (bucketed into powers of two) and "goal words"—specific keywords that the model was required to incorporate into its generation.

To evaluate performance, the researchers compared these fine-tuned T5 models against a massive 137B parameter decoder-only LLM across three datasets: C4, Reddit Writing Prompts (RWP) for creative narrative, and ROC Stories for commonsense reasoning.

Key Insights: Continuation as a Subtask

A central finding of this work is that continuation is essentially a special case of infilling where the right-hand context is empty. Because T5 is trained to handle the more complex task of bidirectional context, it adapts to unidirectional continuation with ease.

Conversely, the study found that even massive 137B parameter models struggle with few-shot infilling. These models often ignore the text following the blank, treating the task as a standard continuation and resulting in poor coherence. This suggests that model scale alone cannot compensate for an architectural inability to process future context during the generation of a middle span.

Results and Controllability

The fine-tuned T5 models demonstrated remarkable "obedience" to user constraints:

  • Length Control: The models successfully hit the requested length bucket over 74% of the time.
  • Keyword Integration: The models incorporated all specified goal words in over 79% of test cases.
  • Human Evaluation: In FitB tasks, human raters significantly preferred the T5 outputs over those from the 137B parameter model. In continuation tasks, while the 137B model held a slight edge due to its sheer size and world knowledge, the T5 models remained highly competitive despite being a fraction of the size.

Furthermore, the models showed strong domain transfer. Even when fine-tuned on general web text, the T5 models performed well on the creative, narrative-driven Reddit Writing Prompts, indicating that the logic of infilling is a transferable linguistic skill.

Conclusion

This research demonstrates that we do not need to maintain separate models for different writing assistance tasks. A single encoder-decoder model, pre-trained on span corruption and fine-tuned with goal conditioning, can provide a versatile and controllable tool for writers. As the field moves toward more interactive AI interfaces, shifting toward architectures that natively support both infilling and continuation will be vital for creating seamless human-AI collaboration.

Summary updated

Abstract

The task of inserting text into a specified position in a passage, known as fill in the blank (FitB), is useful for a variety of applications where writers interact with a natural language generation (NLG) system to craft text. While previous work has tackled this problem with models trained specifically to do the fill-in-the-blank task, a more useful model is one that can effectively perform _both_ FitB and continuation. In this work, we evaluate the feasibility of using a single model to do both tasks. We show that models pre-trained with a FitB-style objective are capable of both tasks, while models pre-trained for continuation are not. Finally, we show how FitB models can be easily finetuned to allow for fine-grained control over the length and word choice of the generation.
BibTeX
@inproceedings{ippolito2022case,
  author = {Daphne Ippolito and Liam Dugan and Emily Reif and Ann Yuan and Andy Coenen and Chris Callison-Burch},
  title = {{The Case for a Single Model that can Both Generate Continuations and Fill in the Blank}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=oklOXVQdVj},
}
Figure