Constrained Language Modeling for Formal Verse

This direction is merged into another direction: constrained creative generation. Its last version stays readable and citable here.

Version 1 of 124 papers30 conceptsfrozen 2026-10-04

Large language models (LLMs) are famously fluent, yet they consistently stumble when asked to write a simple Shakespearean sonnet or a classical Chinese quatrain. While they can mimic the "vibe" of poetry, they frequently fail the math of poetry: the rigid, character-level requirements of meter, rhyme, and syllable counts.

Recent research suggests this failure is not a lack of creativity, but a fundamental mismatch between standard model architectures and the structural demands of formal verse. To bridge this gap, researchers are moving away from general-purpose generation toward systems that treat formal constraints as hard architectural requirements.

The Tokenization Barrier

The primary reason models like GPT-4 struggle with poetry is that they are effectively "blind" to the phonetic and structural building blocks of language. Standard LLMs process text using subword tokens—arbitrary clusters of characters—rather than individual letters or sounds.

As Jonas Belouadi and Steffen Eger argue in their work on ByGPT5, this makes tasks like rhyming or counting syllables nearly impossible because the model cannot "see" the internal structure of the words it is generating. To solve this, researchers are increasingly adopting "token-free" or byte-level architectures. For example, Chengyue Yu and colleagues developed CharPoet, which operates character-by-character to achieve near-perfect format accuracy in Chinese poetry, drastically outperforming much larger token-based models.

Planning vs. Probabilistic Guessing

Standard generation is a "left-to-right" probabilistic process: the model predicts the next word based on what came before. However, formal poetry often requires working backward from a constraint.

In Adapting a Sonnet Model to Chinese Poetry, Zhuohan Xie, Trevor Cohn, and Jey Han Lau demonstrated that reversing the generation order—writing from right-to-left—allows a model to better capture complex rhyming patterns. Similarly, the LYRA system developed by Yufei Tian and others uses a "plan-and-write" framework. By separating the structural template (the "plan") from the linguistic content (the "write"), these systems ensure that the rhythmic requirements of a melody or meter are satisfied before a single word is chosen.

Explicit Constraints and Hybrid Systems

When implicit learning from large datasets fails to produce consistent results, researchers turn to explicit enforcement. The Mechanical Bard, created by Edwin Agnew and his team, combines the creative fluency of GPT-2 with human-curated linguistic templates. This hybrid approach ensures that the model cannot "hallucinate" its way out of a rhyme scheme.

This direction extends beyond poetry into other highly regulated domains. In SS-GEN, Yi Feng and colleagues apply similar constraint-based logic to clinical "Social Stories" for children with ASD, where the ratio of sentence types is strictly mandated. Whether the goal is a Sanskrit verse in Chandomitra or a professional Hebrew diacritization in Nakdan, the emerging consensus is clear: for text with rigid rules, probabilistic fluency must be anchored by structural enforcement.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_constrained_language_modeling_for_formal_ea942e5d,
  title = {Constrained Language Modeling for Formal Verse},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/constrained-language-modeling-for-formal-verse/rdr_ea942e5dd2e6430ca400ec44beef67ff},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_constrained_language_modeling_for_formal_ea942e5d_v1,
  title = {Constrained Language Modeling for Formal Verse},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/constrained-language-modeling-for-formal-verse/rdr_ea942e5dd2e6430ca400ec44beef67ff/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}