PoeLM: A Meter- and Rhyme-Controllable Language Model for Unsupervised Poetry Generation

Aitor Ormazabal, Mikel Artetxe, Manex Agirrezabal, Aitor Soroa, Eneko Agirre

TL;DR

PoeLM is a novel framework for generating formal verse poetry—complete with strict meter and rhyme—without ever training on a single poem. By treating regular prose as "unstructured poetry" and annotating it with structural control codes, the authors train a Transformer that can be prompted to "plan" its writing to meet specific poetic constraints.

What Problem Does This Paper Address?

Writing formal verse poetry (like sonnets) is a high-wire act for AI. It requires adhering to a syllabic meter (a fixed number of syllables per line) and a rhyme scheme (specific ending sounds).

Most existing poetry generators rely on supervised learning, meaning they need large datasets of existing poems to learn these patterns. This creates two major hurdles:

  1. Data Scarcity: For many languages and specific poetic forms, there simply isn't enough digitized data to train a robust model.
  2. Lack of Control: Standard large language models (LLMs) are "black boxes" regarding structure; you can't easily tell GPT-4 to ensure the 14th syllable of every second line rhymes with the 14th syllable of the fourth.

Methods

The authors propose an unsupervised approach based on the insight that all text has an implicit structure. A sentence in a newspaper has a specific syllable count and ends in a specific sound, even if it wasn't intended to be "poetic."

1. Structure-Aware Training

The researchers took a standard, non-poetic corpus (prose) and automatically annotated it. They split the text into phrases using punctuation as natural stopping points. For each phrase, they prepended control codes describing its length (<LEN:X>) and its end rhyme sound (<END:sound>).

As shown in Figure 2, a Transformer language model is then trained on this augmented corpus. It learns the relationship between these leading "structure descriptors" and the text that follows.

Figure 2: Proposed method. (a) During training, we split non-poetic text into phrases according to punctuation marks, prepend control codes describing the length and end rhyme of each phrase, and train a transformer language model on it. (b) During inference, we build a structure descriptor with control codes for the desired meter and rhyme scheme, and condition our language model on them to generate formal verse poetry.

2. The Inference Pipeline

To generate a poem, the user provides a "structure descriptor"—a sequence of control codes representing the desired meter and rhyme scheme. The generation happens in three steps:

  • Candidate Generation: The model generates 3,000 potential completions based on the structure descriptor.
  • Filtering: A script throws away any candidates that failed the strict constraints (e.g., wrong syllable count or accidental word repetition).
  • Re-ranking: The remaining valid poems are scored for fluency by the LM (this time without the structure codes) to pick the most natural-sounding result.

Datasets

  • Training: The authors used EusCrawl for Basque (2.5GB) and a massive 700GB subset of mC4 for Spanish.
  • Evaluation: To test the model's performance against real poetry, they used the DISCO dataset (Spanish sonnets) and the BDB dataset (Basque poems).
  • Tools: Syllabification and rhyme detection were handled by rule-based finite-state transducers (Foma).

Key Insights and Ideas

  • Prose as "Latent" Poetry: The most powerful idea here is that you don't need poems to learn the mechanics of poetry. By forcing the model to predict prose given structural constraints, it learns to "plan" its vocabulary choices to hit a target length and sound.
  • The "Rhyme Spike": The authors analyzed the model's perplexity (uncertainty). As the model approaches the end of a line, its predictive advantage over a standard LM spikes (see Figure 3). Because it knows the required rhyme sound in advance, it has much less "uncertainty" about which word comes next compared to a model just guessing the next token.
  • Unsupervised Versatility: Because the model isn't tied to a specific poetic dataset, it can generate any structure—from a classic sonnet to a completely invented meter—just by changing the initial control codes.

Key Results

  • Validity: While a standard baseline LM failed to generate a single valid poem, PoeLM produced valid Spanish poems 30.9% of the time and Basque poems 23.4% of the time before filtering.
  • Human Preference: In head-to-head rankings, human evaluators preferred PoeLM’s generated poems over those written by renowned poets in 37.3% of cases and over laymen volunteers in 39.3% of cases.
  • Fluency: The model was ranked in the top two positions (out of three) in 57.3% of human evaluation trials, suggesting its output is highly competitive with human-authored verse.

Limitations and Future Work

  • Language Constraints: The method relies on high-quality automated syllabification and rhyme detection. This is straightforward for phonetic languages like Spanish or Basque but significantly harder for English, where spelling and pronunciation are often at odds (e.g., "though" vs. "tough").
  • Poetic Devices: The current Spanish system does not account for synalephas (merging vowels across word boundaries), a common technique in Spanish poetry.
  • Future Directions: The authors aim to extend the framework to control other attributes beyond just meter and rhyme, such as the topic or emotional tone of the poem.

Summary updated

Abstract

Formal verse poetry imposes strict constraints on the meter and rhyme scheme of poems. Most prior work on generating this type of poetry uses existing poems for supervision, which are difficult to obtain for most languages and poetic forms. In this work, we propose an unsupervised approach to generate poems following any given meter and rhyme scheme, without requiring any poetic text for training. Our method works by splitting a regular, non-poetic corpus into phrases, prepending control codes that describe the length and end rhyme of each phrase, and training a transformer language model in the augmented corpus. During inference, we build control codes for the desired meter and rhyme scheme, and condition our language model on them to generate formal verse poetry. Experiments in Spanish and Basque show that our approach is able to generate valid poems, which are often comparable in quality to those written by humans.
BibTeX
@inproceedings{ormazabal2022poelm,
  author = {Aitor Ormazabal and Mikel Artetxe and Manex Agirrezabal and Aitor Soroa and Eneko Agirre},
  title = {{PoeLM: A Meter- and Rhyme-Controllable Language Model for Unsupervised Poetry Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.268},
}
Figure