Bidirectional Context for Text Infilling

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 183 papers105 conceptsfrozen 2026-10-04

Most large language models (LLMs) are "causal"—they predict the next word based solely on what came before. While this works well for generating a story from scratch, it fails in the common human task of editing: filling a gap in an existing sentence. When you try to insert a phrase into the middle of a paragraph, the words that follow the cursor are just as important as the words that precede it.

Research into text infilling (or "fill-in-the-middle") focuses on architectures and training objectives that allow models to look both ways. By treating the text following a gap as a hard constraint rather than an unknown future, these models achieve a level of semantic and syntactic cohesion that standard autoregressive models cannot match.

The Blind Spot of One-Way Models

Standard autoregressive models like GPT-4 generate text by maximizing the probability of the next token given a prefix. However, as Wanrong Zhu and Eric P. Xing argue in their foundational work on Text Infilling, generating text to fit between two known segments requires simultaneous awareness of both. Without the "suffix" context, a model might generate a perfectly fluent sentence that nonetheless crashes into the following text with a grammatical or logical contradiction.

This is particularly evident in character-level tasks. Houxing Ren and colleagues found that standard LLMs struggle with "sub-tokens"—partial words created when a gap is placed randomly. Their framework, FIM-SE, solves this by using boundary-constrained prompts that explicitly include partial lines from both sides of the gap, ensuring the generated sequence integrates seamlessly with the existing fragments Empowering Character-level Text Infilling by Eliminating Sub-Tokens.

Architectures for Looking Both Ways

To solve the infilling problem, researchers have moved toward architectures that natively support bidirectional attention.

Encoder-Decoder Superiority

While decoder-only models are the current industry standard, Maria Becker and Siting Liang demonstrate that encoder-decoder architectures (like BART) are fundamentally better at "reading between the lines." Because the encoder can process the full context of both surrounding statements before the decoder begins, the model can reconstruct the implicit background knowledge that connects them Reconstructing Implicit Knowledge with Language Models.

Discrete Diffusion and Non-Autoregressive Models

A more radical departure is the use of discrete diffusion models. Unlike traditional models that generate one word at a time, diffusion models like DDCap refine an entire sequence simultaneously. Zheng Zhang and Zhe Gan show that this "all-at-once" approach is naturally suited for inpainting, as the model considers future and past tokens as equal constraints during the denoising process Exploring Discrete Diffusion Models for Image Captioning. Similarly, the SUNDAE model by Nikolay Savinov uses step-unrolled denoising to iteratively refine text, breaking the left-to-right bottleneck Step-unrolled Denoising Autoencoders for Text Generation.

The Crossword Intuition

A helpful way to visualize this research is the "crossword puzzle" framing proposed by Junyu Luo. In a crossword, you don't just guess a word based on the clue; you use the intersecting letters from other words. Their Document Embedding Network (DEN) treats text reconstruction as a bidirectional puzzle where local context and global themes must align to resolve ambiguity Learning Document Embeddings with Crossword Prediction.

This "future-hinting" is now being used to train better models. Siyan Zhao and Bo Liu introduced IGPO, which uses the inpainting capabilities of diffusion LLMs to inject "ground-truth hints" during training. By filling in the gaps between a current state and a correct solution, the model learns logical steps that a unidirectional model would likely miss Inpainting-Guided Policy Optimization for Diffusion Large Language Models.

Applications: From Poetry to Privacy

The ability to infill text has transformed several specialized domains:

  • Creative Writing: The BIPro framework by Xu Zou uses block-based attention to generate traditional Chinese poetry. By mimicking the human process of drafting and revision—where a poet might write the end of a couplet before the middle—BIPro outperforms much larger models like GPT-4 BIPro: Zero-shot Chinese Poem Generation.
  • Writing Assistants: Tools like Effidit from Shuming Shi and the IGA assistant from Simeng Sun move beyond simple autocomplete. They analyze the text after the cursor to provide contextually appropriate phrase completions and rhetorical goal-setting Effidit: Your AI Writing Assistant.
  • Privacy and Security: Bidirectional context is a double-edged sword. Arpita Vats shows that LLMs can "recover" sensitive data that has been masked for privacy by using surrounding context to predict the hidden tokens Recovering from Privacy-Preserving Masking with Large Language Models. This is taken further by Wenlong Meng in the R.R. (Recollect and Rank) attack, which uses bidirectional context to deanonymize "scrubbed" datasets with high accuracy R.R.: Unveiling LLM Training Privacy through Recollection and Ranking.

Why It Works: The Power of Constraints

The core intuition behind the success of bidirectional infilling is that the suffix acts as a boundary condition. In a standard autoregressive model, the "search space" for the next word is vast. In an infilling task, the search space is constrained on both sides.

As Yujia Xie and Pengcheng He note in their work on iterative summarization, incorporating suffix context ensures that new insertions transition naturally into existing content, preventing the disjointedness common in prefix-only generation Interactive Editing for Text Summarization. By treating the generation task as a bridge between two known points, bidirectional models achieve a level of structural integrity that "blind" left-to-right generation simply cannot replicate.

Papers in this version

Showing 60 of 83 papers.

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_bidirectional_context_for_text_infilling_8c2b2ee9,
  title = {Bidirectional Context for Text Infilling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bidirectional-context-for-text-infilling/rdr_8c2b2ee920ba40969f9d064ae1b54ae5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_bidirectional_context_for_text_infilling_8c2b2ee9_v1,
  title = {Bidirectional Context for Text Infilling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bidirectional-context-for-text-infilling/rdr_8c2b2ee920ba40969f9d064ae1b54ae5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}