TL;DR
This paper argues that whitespace, often overlooked in NLP, is a crucial element in poetry that significantly impacts meaning and structure. The authors introduce a new typology for analyzing whitespace, WISP, and demonstrate its importance by comparing whitespace usage in published poems, unpublished poems, and LLM-generated poems, highlighting challenges for LLMs in accurately representing poetic layout.
What Problem Does This Paper Address?
In natural language processing, whitespace is typically treated as a mere separator or is stripped entirely during preprocessing. However, for poetry, whitespace is a fundamental artistic and semantic element, influencing rhythm, meaning, and visual structure. This paper addresses the lack of attention to whitespace in NLP research, particularly concerning its role in poetic form and the challenges it poses for Large Language Models (LLMs). The authors aim to bridge the gap between computational analysis and the nuanced artistic use of whitespace in poetry, highlighting that LLMs, despite their advanced capabilities, struggle to accurately preserve and interpret this critical feature.
Methods
The authors developed a novel typology called Whitespace In Spatial Poetics (WISP) to categorize different uses of whitespace in poetry. WISP distinguishes between five categories: line breaks, prefix space, internal space, vertical space, and line length. They then employed a multi-pronged approach:
- Data Collection: They gathered three datasets: published poems from the Poetry Foundation, unpublished poems from Reddit's r/OCPoetry, and LLM-generated poems using GPT-4 and Sonnet 3.7.
- Linearization Systems: To handle the complexities of web-scraped poetry data, they tested various HTML-to-text conversion methods (resiliparse, trafilatura, jusText) and developed a custom pipeline, "WISP-ify," designed to better preserve poetic formatting. They also explored image-to-text models for poem screenshots.
- Human Evaluation (WISP-Bench): A crucial part of their methodology was a human evaluation framework, WISP-Bench, designed to assess how well different linearization methods preserve specific whitespace properties. This benchmark uses a three-tiered system: Presence Match, Fuzzy Match, and Exact Match for structural, relational, and absolute fidelity, respectively.
- Computational Analysis: They conducted large-scale analyses of whitespace usage across their datasets, examining variations by poetic form, time period, and topic. They also analyzed dependency triples spanning line breaks to understand syntactic structures related to whitespace.
Datasets
- Published Poems: 19,457 poems from the Poetry Foundation website.
- Unpublished Poems: 11,984 poems from the r/OCPoetry subreddit.
- LLM-Generated Poems: Approximately 51,400 poems generated by GPT-4 (OpenAI) and Sonnet 3.7 (Anthropic).
- New Benchmark: The authors also released a subset of 2,800 public-domain poems with preserved formatting to facilitate future research.
Key Insights and Ideas
- Whitespace is Not Trivial: The study strongly reinforces that whitespace is not merely decorative but an integral part of poetic meaning, structure, and rhythm, especially in modern and free verse poetry.
- LLMs Struggle with Poetic Whitespace: Despite advancements, LLMs often fail to accurately preserve or creatively utilize poetic whitespace, especially non-standard forms like prefix and internal spacing. This highlights a significant gap in their understanding of poetic nuance.
- Linearization Matters: The choice of HTML-to-text conversion method significantly impacts the preservation of whitespace. Standard NLP preprocessing pipelines are often inadequate for poetic data.
- Poetic Form Influences Whitespace: Different poetic forms exhibit distinct whitespace patterns. For instance, free verse shows more variation and prefix spacing, while couplets tend to have more internal space.
- Historical Trends: Whitespace usage in poetry has generally increased over time, particularly in the 20th century, reflecting a growing emphasis on visual and spatial elements.
- Topic Correlation: Certain topics, like "Gender-Sexuality" and "The Body," are associated with higher whitespace usage, suggesting a link between thematic content and spatial expression.
Key Results
- Dataset Comparison: Published poems showed more creative/non-standard whitespace (especially prefix spacing) than Reddit poems. LLMs, when prompted for creative whitespace, produced poems with more prefix whitespace than published free-verse poems.
- Linearization Performance: The custom WISP-ify pipeline and the resiliparse tool performed best in human evaluations for preserving whitespace, significantly outperforming jusText and trafilatura. LLMs also showed mixed results, with similar strengths and weaknesses in preserving line breaks versus prefix/internal spacing.
- LLM Generation: LLMs, when not explicitly prompted for creative whitespace, rarely produced poems with non-standard prefix spacing. However, they demonstrated the capability to generate whitespace-heavy poems when instructed.
- Punctuation at Line Breaks: Commas are the most common punctuation at line ends across forms, but colons and semicolons are more likely at line ends in couplets and common measure. Free verse poems generally have less frequent punctuation at line breaks.
Limitations and Future Work
- Language and Script Specificity: The analysis is limited to English-language poems in the Roman script and may not generalize to other languages or scripts.
- Digital Presence Bias: Conclusions about historical trends are based on poets' digital presence on the Poetry Foundation, potentially overrepresenting North American poets and limiting representativeness.
- LLM Memorization: LLMs might "memorize" copyrighted text from training data, potentially influencing generation even with explicit instructions not to reuse original text.
- OCR Errors in Books: Poems from books, even if digitized, may suffer from whitespace issues due to OCR errors, a problem not fully addressed in this study.
- Future Work: The authors suggest future work should focus on multimodal models that can better handle visual layout, including CSS styling and JavaScript. They also emphasize the need for more robust linearization methods specifically designed for poetic texts and the potential for LLMs to assist in tasks like encoding poetry using TEI.
Summary updated