TL;DR
PoeLM is a novel framework for generating formal verse poetry—complete with strict meter and rhyme—without ever training on a single poem. By treating regular prose as "unstructured poetry" and annotating it with structural control codes, the authors train a Transformer that can be prompted to "plan" its writing to meet specific poetic constraints.
What Problem Does This Paper Address?
Writing formal verse poetry (like sonnets) is a high-wire act for AI. It requires adhering to a syllabic meter (a fixed number of syllables per line) and a rhyme scheme (specific ending sounds).
Most existing poetry generators rely on supervised learning, meaning they need large datasets of existing poems to learn these patterns. This creates two major hurdles:
- Data Scarcity: For many languages and specific poetic forms, there simply isn't enough digitized data to train a robust model.
- Lack of Control: Standard large language models (LLMs) are "black boxes" regarding structure; you can't easily tell GPT-4 to ensure the 14th syllable of every second line rhymes with the 14th syllable of the fourth.
Methods
The authors propose an unsupervised approach based on the insight that all text has an implicit structure. A sentence in a newspaper has a specific syllable count and ends in a specific sound, even if it wasn't intended to be "poetic."
1. Structure-Aware Training
The researchers took a standard, non-poetic corpus (prose) and automatically annotated it. They split the text into phrases using punctuation as natural stopping points. For each phrase, they prepended control codes describing its length (<LEN:X>) and its end rhyme sound (<END:sound>).
As shown in Figure 2, a Transformer language model is then trained on this augmented corpus. It learns the relationship between these leading "structure descriptors" and the text that follows.
2. The Inference Pipeline
To generate a poem, the user provides a "structure descriptor"—a sequence of control codes representing the desired meter and rhyme scheme. The generation happens in three steps:
- Candidate Generation: The model generates 3,000 potential completions based on the structure descriptor.
- Filtering: A script throws away any candidates that failed the strict constraints (e.g., wrong syllable count or accidental word repetition).
- Re-ranking: The remaining valid poems are scored for fluency by the LM (this time without the structure codes) to pick the most natural-sounding result.
Datasets
- Training: The authors used EusCrawl for Basque (2.5GB) and a massive 700GB subset of mC4 for Spanish.
- Evaluation: To test the model's performance against real poetry, they used the DISCO dataset (Spanish sonnets) and the BDB dataset (Basque poems).
- Tools: Syllabification and rhyme detection were handled by rule-based finite-state transducers (Foma).
Key Insights and Ideas
- Prose as "Latent" Poetry: The most powerful idea here is that you don't need poems to learn the mechanics of poetry. By forcing the model to predict prose given structural constraints, it learns to "plan" its vocabulary choices to hit a target length and sound.
- The "Rhyme Spike": The authors analyzed the model's perplexity (uncertainty). As the model approaches the end of a line, its predictive advantage over a standard LM spikes (see Figure 3). Because it knows the required rhyme sound in advance, it has much less "uncertainty" about which word comes next compared to a model just guessing the next token.
- Unsupervised Versatility: Because the model isn't tied to a specific poetic dataset, it can generate any structure—from a classic sonnet to a completely invented meter—just by changing the initial control codes.
Key Results
- Validity: While a standard baseline LM failed to generate a single valid poem, PoeLM produced valid Spanish poems 30.9% of the time and Basque poems 23.4% of the time before filtering.
- Human Preference: In head-to-head rankings, human evaluators preferred PoeLM’s generated poems over those written by renowned poets in 37.3% of cases and over laymen volunteers in 39.3% of cases.
- Fluency: The model was ranked in the top two positions (out of three) in 57.3% of human evaluation trials, suggesting its output is highly competitive with human-authored verse.
Limitations and Future Work
- Language Constraints: The method relies on high-quality automated syllabification and rhyme detection. This is straightforward for phonetic languages like Spanish or Basque but significantly harder for English, where spelling and pronunciation are often at odds (e.g., "though" vs. "tough").
- Poetic Devices: The current Spanish system does not account for synalephas (merging vowels across word boundaries), a common technique in Spanish poetry.
- Future Directions: The authors aim to extend the framework to control other attributes beyond just meter and rhyme, such as the topic or emotional tone of the poem.
Summary updated