TL;DR
This paper introduces a novel task for generating metaphorical poetry, revealing that even human writers struggle to create coherent metaphorical connections, with only 24% of poems successfully conveying the intended metaphor. The authors also release a dataset of these poems and categorize common failure modes, offering insights for computational approaches to figurative language generation.
What Problem Does This Paper Address?
Generating truly meaningful and creative text, especially poetry, remains a significant challenge for computational systems. While progress has been made in generating poetic structures and sounds, ensuring the conceptual and emotional depth of poetry is still an open problem. Existing work often produces text that is ambiguous or lacks a clear, intentional message. This paper tackles the specific challenge of generating meaningful figurative language, focusing on metaphors, which are a cornerstone of poetic expression. The authors aim to understand the difficulties involved in creating coherent metaphorical connections and to establish a baseline for human performance on this task, which can then inform the development of computational models.
Methods
The core of this paper is a carefully designed poetry writing task. The authors recruited 200 participants from Amazon Mechanical Turk and assigned them one of ten randomly generated metaphorical prompts. These prompts were created by pairing concrete nouns with poetic themes (e.g., "anger is wood," "surrender is a book"), drawing from existing lists. Participants were asked to write a four-line poem that expressed the given metaphor without using the exact words of the prompt, while still incorporating poetic elements like rhyme and line breaks.
To evaluate the success of these poems, the authors developed a categorization scheme. Poems were assessed for whether they successfully conveyed the given metaphor. For poems that failed, they were further categorized into:
- Off-Topic: Fails to include aspects of either word in the metaphor.
- No Connection: Explores only one concept or fails to relate the two conceptual domains.
- Attributional Connection: Attributes the abstract concept directly to the concrete noun (e.g., "the bench is angry").
- Offset Connection: Expresses a shared feature between a word in the metaphor and a related word, but not directly the given metaphor (e.g., "anger is fire" when the prompt was "anger is wood").
- Incoherent Connection: Relates the two words unclearly.
- Metaphorical Connection: Successfully relates the two words metaphorically as intended.
The authors and a second evaluator independently assessed the poems, achieving high agreement (97%) on both metaphorical success and off-topic classification.
Datasets
The paper introduces a newly created dataset of 186 poems, each associated with metadata including the prompt metaphor and its categorization (e.g., metaphorical connection, no connection, etc.). This dataset is derived from the poetry writing task described above and is made publicly available. It includes both successful and unsuccessful attempts at metaphorical connections, serving as a benchmark for future research.
Key Insights and Ideas
- Human performance is surprisingly low: The most striking insight is that even with clear prompts and instructions, only 24% of human writers successfully created a poem that conveyed the intended metaphorical connection. This highlights the inherent difficulty of generating meaningful figurative language.
- "Sensible" connections are a trap: The authors found that poems often made "sensible" but non-metaphorical connections (e.g., "the rose died" for "death is a rose"). This suggests that writers may prioritize a plausible connection over a truly metaphorical one, a pitfall that computational systems might also face.
- Word similarity isn't a strong predictor of success: Contrary to intuition, the study found little correlation between the semantic similarity of words in the prompt (measured by Word2Vec) and the success rate of the poems. This implies that successful metaphorical connections are often creative and novel, rather than relying on pre-existing, easily accessible associations.
- Categorizing failure is crucial: The detailed categorization of poem failures (off-topic, no connection, attributional, offset, incoherent) provides valuable insights into why metaphorical connections are difficult to make. This taxonomy is essential for understanding common pitfalls and guiding the development of more effective computational models.
Key Results
- Success Rate: Only 24% of the 186 analyzed poems successfully made the intended metaphorical connection.
- Most Common Failure: The most frequent failure mode was "No Connection" (41% of poems), where the poem either focused on only one concept or failed to link the two domains.
- Other Failure Modes: Attributional connections accounted for 17%, offset connections for 15%, incoherent connections for 15%, and off-topic poems for 7%.
- Prompt Variability: Success rates varied significantly across prompts, from a low of 5% for "surrender is a book" to a high of 47% for "hate is a mist."
- High Evaluator Agreement: The authors achieved 97% agreement between evaluators on whether a poem successfully made the metaphorical connection and whether it was off-topic.
- Task Duration: On average, participants took 13.6 minutes to complete the writing task.
Limitations and Future Work
Limitations: * The study acknowledges that some poems could be ambiguous, leading to evaluator disagreement on categorization. * The reliance on Word2Vec for semantic similarity might not fully capture human associations.
Future Work: * Computational Evaluation: Developing computational methods to evaluate whether a poem expresses a given metaphor, potentially using techniques like conceptual metaphor theory. * Computational Generation: Extending existing generative models to produce poems based on specific metaphorical constraints, rather than just themes. * Computational Aids: Designing tools to assist human writers in generating metaphorical connections, perhaps by suggesting possible links or identifying common errors. * Broader Applications: Recognizing that understanding and generating metaphorical connections has implications beyond poetry, extending to fields like science education and product design.
Summary updated