Lexically Constrained Neural Text Generation

This direction is merged into another direction: fluency-control trade-off. Its last version stays readable and citable here.

Version 1 of 117 papers22 conceptsfrozen 2026-10-04

Neural language models are remarkably fluent, but they are notoriously difficult to steer. In tasks like machine translation or image captioning, we often need the model to include specific terminology—such as a brand name, a technical term, or a detected object—that it might otherwise ignore in favor of more frequent, generic words.

The challenge of lexical constraints is ensuring these required words appear in the output without destroying the sentence's grammatical flow or making the generation process prohibitively slow. Researchers have converged on two primary ways to solve this: modifying the "search" at the end of the process, or teaching the model to handle "hints" from the very beginning.

The most direct way to ensure a word appears in the output is to bake it into the decoding algorithm. In standard beam search, a model explores the most likely sequences of words. To enforce constraints, we can use Constrained Beam Search (CBS).

As demonstrated in Guided Open Vocabulary Image Captioning with Constrained Beam Search by Peter Anderson et al., this approach treats constraints as a "checklist." The search algorithm tracks which required words have been generated and prioritizes sequences that satisfy the remaining constraints. This allows a model to describe objects it may never have seen in its training data by forcing the inclusion of specific image tags.

However, rigid constraints can lead to "disjointed" text. If you force the word "run" into a sentence that requires "running," the grammar breaks. To fix this, Zhongyang Li et al. introduced Disjunctive Positive Constraint (DPC) decoding in Guided Generation of Cause and Effect. Instead of forcing a single exact string, DPC allows the model to choose from a set of morphological variants (e.g., "run," "runs," "running"). This "disjunctive" approach lets the model pick the version that best fits the local grammatical context, bridging the gap between rigid logic and natural fluency.

Soft Constraints through Training and Memory

While decoding-time constraints are effective, they can be computationally expensive and sometimes "bully" the model into awkward phrasing. A more recent alternative is to treat constraints as soft suggestions provided during the encoding stage.

One approach is to change how the model is trained. In Code-Switching for Enhancing NMT with Pre-Specified Translation, Kai Song et al. suggest a data augmentation strategy. By replacing source words with their target-language translations during training (a form of code-switching), the model learns to "fill in the blanks" around pre-specified terms. This teaches the model to naturally adapt its surrounding grammar to the required words, rather than having those words forced in later.

Taking this a step further, Dan Liu et al. proposed LCNMT in Learning Efficient Lexically-Constrained Neural Machine Translation with External Memory. Instead of modifying the search, they integrated an external memory buffer into the Transformer encoder. This memory acts as a "scratchpad" of required terms that the model can attend to while generating.

The intuition here is shift:

  • Decoding constraints are like a supervisor standing over a writer, pointing at a list and saying, "You must use these words now."
  • Encoding constraints are like giving the writer a list of keywords before they start. The writer keeps the keywords in their peripheral vision and weaves them into the narrative where they fit best.

Why the Shift Matters

The evolution of this field shows a move from hard enforcement to contextual integration. Early methods like CBS ensure 100% constraint satisfaction but can be slow and rigid. Newer methods, like those by Dan Liu and Kai Song, prioritize efficiency and fluency. By treating constraints as part of the input context rather than a post-hoc requirement, these models produce translations that feel less like a "forced" output and more like a cohesive thought that happens to include the necessary terms.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_lexically_constrained_neural_text_genera_cb139cf5,
  title = {Lexically Constrained Neural Text Generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lexically-constrained-neural-text-generation/rdr_cb139cf516b948759ba2360a7ed36a67},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_lexically_constrained_neural_text_genera_cb139cf5_v1,
  title = {Lexically Constrained Neural Text Generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lexically-constrained-neural-text-generation/rdr_cb139cf516b948759ba2360a7ed36a67/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}