Introduction
In the current landscape of AI-assisted writing, a functional divide exists between different types of generative tasks. Traditionally, if a writer needs an AI to finish a sentence, they utilize an auto-regressive "continuation" model like GPT. Conversely, if the task involves inserting a missing word or phrase into the middle of a paragraph, a specialized "infilling" model is required. This research, presented at NAACL 2022 by researchers from the University of Pennsylvania and Google Research, challenges this binary approach. The authors propose a "Swiss Army Knife" model: a single architecture capable of handling both continuation and fill-in-the-blank (FITB) tasks while offering fine-grained control over length and content.
Methodology
The researchers focused on the T5 (Text-to-Text Transfer Transformer) architecture. Unlike decoder-only models that primarily process information in one direction, T5’s encoder-decoder structure allows it to simultaneously consider context on both sides of a blank.
To achieve a multi-functional model, the team leveraged T5's original "span corruption" pre-training—a process akin to a massive game of Mad Libs—and further finetuned it on a balanced mix of tasks: 50% infilling (blank in the middle) and 50% continuation (blank at the end). To enhance user control, they introduced conditioning signals or "tags":
- Length Control: Directing the model to generate approximately $N$ words.
- Goal Conditioning: Providing specific keywords or phrases that the model must incorporate into the generated text.
Evaluation and Datasets
The models were evaluated against a massive 137B parameter decoder-only baseline (similar to LaMDA) using few-shot prompting. Testing was conducted across three diverse datasets to ensure robustness:
- C4 (Colossal Clean Crawled Corpus): General web text.
- Reddit Writing Prompts (RWP): Creative narrative stories.
- ROC Stories: Five-sentence stories focused on commonsense logical progression.
Results and Discussion
The study revealed that infilling is essentially a "superset" task. A model capable of infilling can perform continuation by simply placing the blank at the end of a sequence. However, standard continuation models struggle with infilling because they cannot effectively "look ahead" to ensure the inserted text fits the subsequent context.
Key findings include:
- Efficiency over Scale: The 800M parameter T5 model outperformed the 137B parameter LLM on FITB tasks, suggesting that specialized architecture is more critical than raw parameter count for these specific writing aids.
- Human Preference: In human evaluations, the finetuned T5 was rated nearly as high as the 137B LLM for continuation and was virtually indistinguishable from the pre-trained version in FITB quality.
- Controllability: The finetuned model demonstrated high "obedience," hitting target length buckets 74% of the time and incorporating requested goal words in 79% of instances. While pre-trained T5 is naturally good at filling blanks, it tends to produce very short responses; finetuning was essential for following complex user instructions.
Conclusion
This research demonstrates that a single, moderately sized encoder-decoder model can serve as a versatile tool for various writing tasks. By unifying continuation and infilling into one model, developers can reduce resource overhead while providing writers with more flexible and controllable AI partners. Future work may expand these findings to technical domains like legal or medical writing and explore deeper integration into real-time collaborative interfaces.
Summary updated