Formal Structural Constraints in Creative Text Generation

118 concepts85 papers1 proposal

Large language models excel at generating fluent, contextually rich prose. However, if you ask a state-of-the-art model to write a Shakespearean sonnet in perfect iambic pentameter or a traditional Chinese quatrain with strict tonal patterns, it will surprisingly struggle.

As Edwin Agnew observes in Can a Computer Really Write Poetry?, simply scaling up a language model does not solve this issue. Larger models often prioritize semantic fluidity and prose-like naturalness over technical rules. To understand why this happens—and how researchers are fixing it—we have to look at the fundamental tension between probabilistic text generation and discrete structural constraints.

The Autoregressive Bottleneck

Standard language models generate text autoregressively, predicting the next token based on the probability distribution of preceding tokens. This left-to-right approach works beautifully for open-ended generation, but it creates a severe bottleneck when hard, non-local constraints are introduced.

For example, a rhyme scheme requires the final word of a line to match the sound of a word several tokens back. A metrical constraint requires a precise sequence of stressed and unstressed syllables across an entire line. When generating token-by-token, a model cannot easily "look ahead" to ensure that its current lexical choices will perfectly align with a mathematical constraint down the line.

As Jey Han Lau demonstrates in Deep-speare: A joint neural model of poetic language, meter and rhyme, forcing a standard neural network to implicitly learn these multi-layered constraints from raw data creates an intense optimization conflict: maximizing semantic meaning often causes structural violations, and vice versa.

Hybrid Architectures: Marrying Probability and Logic

To overcome this bottleneck, one prominent direction bridges the gap between neural fluidity and symbolic precision by pairing language models with exact constraint engines.

Instead of hoping a model will happen to generate the correct number of syllables, researchers treat generation as a combinatorial optimization problem. For instance, Alexandre Bonlarron et al. in Intertwining CP and NLP: The Generation of Unreasonably Constrained Sentences combine Constraint Programming (CP) with large language models. The symbolic solver enforces hard, binary rules (such as exact character or syllable counts), while the language model ensures the output remains grammatically fluent.

A similar logic applies to finite-state methods. In Generating Topical Poetry, Marjan Ghazvininejad et al. utilize Finite-State Acceptors (FSAs) to rigidly police the rhythmic meter of a sonnet, filtering the output vocabulary at each step so that the recurrent neural network can only select words that preserve the iambic pentameter.

[Language Model] ---> Proposes fluent, topically relevant words
                             |
                             v
[Symbolic Filter] -> Rejects words violating rhyme/meter/syllable rules
                             |
                             v
                      Valid Poetic Line

Explicit Conditioning and Structural Templates

Another way to prevent a model from drifting away from structural rules is to make those rules an explicit part of the input space. Rather than relying on the model to maintain an internal representation of the poetic form, the structure is provided as an external template.

In Generating Chinese Ci with Designated Metrical Structure, Richong Zhang et al. employ a Conditional Variational Auto-Encoder (CVAE) that explicitly encodes varying line lengths, tonal requirements, and rhyming rules as conditioning vectors. By feeding these structural blueprints directly into the neural architecture, the model achieves near-perfect adherence to highly irregular classical formats without sacrificing semantic variety.

Breaking the Left-to-Right Paradigm

Perhaps the most intuitive shift in recent work is moving away from autoregressive generation entirely. Human poets rarely write a poem perfectly from left to right in a single pass; they draft, look ahead to the rhyme, and iteratively polish their lines.

One approach is to reverse the generation order. Because the rhyme word at the end of a line is the most constrained element, models can generate text from right to left, anchoring the structure around the rhyme before filling in the preceding context.

Alternatively, researchers are utilizing non-autoregressive diffusion models to manipulate entire sentences simultaneously. Zhiyuan Hu et al. introduce this concept in PoetryDiffusion: Towards Joint Semantic and Metrical Manipulation in Poetry Generation. By refining all tokens in a verse in parallel, a diffusion framework allows semantic meaning and metrical constraints to shape each other dynamically over multiple denoising steps. The model never paints itself into a structural corner, because it can adjust the beginning of a line to accommodate a rigid constraint at the end.

Key Foundational Work

The Linguistic Basis of Text Generation (1987) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_constrained_creative_generation_bdbc75db,
  title = {constrained creative generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/constrained-creative-generation/rdr_bdbc75dbf63840fbaa0943a91ff1848c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_constrained_creative_generation_bdbc75db_v1,
  title = {constrained creative generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/constrained-creative-generation/rdr_bdbc75dbf63840fbaa0943a91ff1848c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake