Variable-Length Span Infilling for Generative Text Editing

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 117 papers24 conceptsfrozen 2026-10-04

Standard language models like GPT-2 are traditionally "left-to-right" machines, excelling at predicting the next word but struggling when asked to fill a hole in the middle of an existing sentence. Conversely, masked models like BERT can look at the surrounding context but are often constrained by a one-to-one mapping, where one mask equals exactly one predicted token. A growing body of research has converged on variable-length span infilling—a technique that allows models to replace a single mask with a sequence of any length, effectively unifying understanding and generation.

The Mechanism of the Elastic Mask

The core intuition behind this research direction is moving away from "fixed-length" substitution. In traditional masked language modeling, if you mask three words, the model knows it must find exactly three words to fit. This creates a "length leakage" that simplifies the task too much.

As Mike Lewis and colleagues demonstrated with BART, replacing an entire contiguous span of tokens with a single mask token forces the model to first predict how much information is missing before deciding what that information is. This "span-based" approach prevents the model from relying on trivial positional cues and encourages a deeper grasp of global coherence.

Unifying Architectures via Masking Ratios

By varying the length and frequency of these masks, researchers have found they can bridge the gap between different model families. Kaitao Song and the authors of MASS showed that encoder-only (BERT-style) and decoder-only (GPT-style) objectives are actually two ends of the same spectrum.

  • Short masks emphasize local linguistic understanding.
  • Long masks (up to the full sequence) turn the task into pure generative modeling.

This unification is taken further in the General Language Model (GLM) by Zhengxiao Du and others, which uses "autoregressive blank infilling" to handle NLU and generation tasks within a single architecture. Theoretical work by Qi Zhang suggests that this flexibility in masking ratios is what allows models to overcome "length misalignment," a common failure point where models struggle to generate text that differs significantly in length from their training samples.

From Pre-training to Precise Editing

The ability to "fill in the blanks" with arbitrary lengths has immediate practical applications in controllable text editing. Instead of rewriting a sentence from scratch, models can perform surgical strikes on specific spans.

  • Style Transfer: Aliaksei Severyn and the MASKER team use this to identify style-heavy tokens and replace them with domain-appropriate content of different lengths, ensuring the "meaning" stays while the "flavor" changes.
  • Creative Writing: In SongRewriter, Liangyou Li employs multi-level randomized masking (token, sentence, and document) to allow users to rewrite lyrics while maintaining strict constraints like rhyme and rhythm.
  • Data Augmentation: The Mask-then-Fill framework by Jun Gao and Changlong Yu uses variable-length infilling to create diverse training data for event extraction. By replacing non-essential "adjunct" fragments with new spans, they force models to focus on the core logic of a sentence rather than memorizing specific syntactic patterns.

By treating text generation as a flexible infilling problem, these researchers have moved beyond simple "next-word" prediction toward a more robust, bidirectional understanding of how language fits together.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_variable_length_span_infilling_for_gener_65605d01,
  title = {Variable-Length Span Infilling for Generative Text Editing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/variable-length-span-infilling-for-generative-text-editing/rdr_65605d014f6e491f821e6b0207bafcba},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_variable_length_span_infilling_for_gener_65605d01_v1,
  title = {Variable-Length Span Infilling for Generative Text Editing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/variable-length-span-infilling-for-generative-text-editing/rdr_65605d014f6e491f821e6b0207bafcba/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}