TL;DR
Researchers at Leiden University explored whether Large Language Models (LLMs) can sustain an "infinite" poem under strict formal constraints. By comparing manual co-creation with automated hybrid pipelines—combining vector databases and GPT-4—they found that while AI can generate thousands of couplets, it still struggles with creative repetition and the deep semantic nuance required for truly "poetic" pairings.
What Problem Does This Paper Address?
The rise of generative AI has sparked a "crisis of authorship" in creative writing. While LLMs can churn out vast quantities of text, can they handle the high-precision, lexical-level creativity required for poetry?
The authors tackle this by testing AI against a specific poetic form: the anaphoric litany. Using a seed poem titled "resurrection," they constrained the AI to write couplets where every line begins with the phrase "one more," leaving only the final word of each line to be generated. This "minimalist" setup isolates the AI's ability to create meaningful, surprising, or "poetic" relationships between two words—a task that requires more than just statistical probability.
Methods
The research was divided into two primary experiments:
- Experiment 1 (Manual Prompting): The authors used GPT-4 to generate batches of 100 couplets. The goal was to reach 365 unique couplets (a "year" of poetry). This required heavy human curation and "prompt engineering" to overcome the model's tendency to repeat itself.
- Experiment 2 (Automated Pipelines): To reach a benchmark of 1,000+ couplets, the team built two automated methods:
- Method #1 (GPT + Vector Search): GPT-4 generates the first word; a vector database (ChromaDB using Sentence-BERT embeddings) finds the $n$-th most similar word to serve as the pair.
- Method #2 (Random Seed + GPT): A word is selected at random from a Scrabble dictionary, and GPT-4 is tasked with generating a "poetic" pairing for it.
The results were evaluated subjectively by the lead author (a professional poet) based on whether the pairings were "poetic" enough to be included in the final artwork without editing.
Datasets
- Seed Poem: An original 121-couplet poem by the lead author served as the training/prompting baseline.
- Wordlist: The NASPA Zyzzyva (2019 Collins Scrabble Words) list, containing 279,496 entries, was used to populate the vector database and provide random seeds for Experiment 2.
- Evaluation Set: A sample of 1,000 generated couplets from Method #1 and 250 from Method #2 were manually classified by poetic relationship (rhyme, synonym, antonym, etc.).
Key Insights and Ideas
- The "Ember" Problem: LLMs have "favorite" words. Despite explicit instructions not to repeat end-words, GPT-4 frequently defaulted to a specific aesthetic vocabulary (e.g., ember, gossamer, mosaic, nebula), likely reflecting biases in its training data (19th/20th-century literature).
- Vector Similarity $\neq$ Poetics: Method #1, which relied on mathematical vector similarity, produced mostly "boring" results. It excelled at alliteration and word-form similarity (e.g., de-prefix matching) but failed to capture the "leap" of a poetic metaphor.
- LLMs as "Pairers," not "Seeders": The AI performed significantly better when given a random, non-poetic seed word and asked to find a creative partner (Method #2) than when it was asked to generate the entire couplet from scratch.
- Chain-of-Thought for Art: The authors used "Chain-of-Thought" prompting to ask the AI to explain its poetic rationale. Interestingly, about 18% of the time, the AI could not logically justify the relationship it had generated, highlighting the "involuntary" nature of its output.
Key Results
The quantitative gap between the two automated methods was stark. As shown in the table below, Method #2 (where the LLM handles the pairing) was nearly six times more successful at producing "poetic" results than Method #1 (vector-led pairing).
Qualitatively, Method #2 favored antonyms (~40%) and clichés/idioms (~24%) to establish relationships. While these are standard linguistic tools, the poet-author noted that the AI still lacks the ability to "mock" or subvert these clichés in the way a human writer might (e.g., changing "apples and oranges" to "apples and orangutans").
The project culminated in two public art installations, including a 20-minute animated web poem and a large-scale projection mapping project where users could contribute their own words to a "living poetry mural."
Limitations and Future Work
The authors acknowledge that the evaluation was limited by a single-rater bias (the lead author). Future studies plan to use multiple third-party raters, including other poets and even AI evaluators.
Technically, the authors suspect that LLMs struggle with this task because they operate on tokens, not words. A "word-level" constraint might be fundamentally contrary to how these models "think" in probability landscapes. Future work will explore longer linguistic structures and "multi-modal" interfaces to see if AI can move beyond simple word-pairing into more complex, intentional poetic "worlds of meaning."
Summary updated