Unveiling the Essence of Poetry: Introducing a Comprehensive Dataset and Benchmark for Poem Summarization

Ridwan Mahbub, Ifrad Khan, Samiha Anuva, Md Shahriar, Md. Tahmid Rahman Laskar, Sabbir AhmedEMNLP 2023 · DOI

Introduction

While standard Natural Language Processing (NLP) models have achieved remarkable success in summarizing factual domains like news and finance, they often falter when faced with the creative complexity of literature. Researchers have introduced PoemSum, the first comprehensive dataset and benchmark specifically designed for the abstractive summarization of poetry. This study reveals that current state-of-the-art models struggle significantly with the figurative language and metaphorical depth inherent in verse, often defaulting to literal interpretations or simple extraction rather than capturing a poem's true essence.

The Challenge of Figurative Language

Most text summarization research focuses on "literal" domains where success is measured by the condensation of key facts. Poetry, however, relies on metaphors, personification, and symbolism. A poem’s meaning is rarely found on the surface; for instance, a poet writing about "bleeding" may be conveying emotional anguish rather than a medical event. To achieve true Natural Language Understanding, models must move beyond literal extraction and learn to decipher the figurative patterns that convey a poet's actual intent.

The PoemSum Dataset

The core contribution of this work is the PoemSum dataset, a curated collection of 3,011 samples. Each sample consists of a poem paired with an expert-written summary or interpretation. The dataset is diverse, featuring 930 different poets and varying poem lengths.

Summary of the PoemSum dataset statistics, including poem counts and average lengths.

Summaries were sourced from "Poem Analysis" and underwent strict quality control. The researchers manually verified each entry, rejecting summaries that were purely biographical, focused solely on rhyme schemes, or merely "translated" the poem into simple language without providing deeper analysis.

Benchmarking Methodology

The authors evaluated four state-of-the-art transformer-based models to establish a benchmark:

  • T5 (Text-to-Text Transfer Transformer): An encoder-decoder model treating all tasks as string-to-string mappings.
  • BART: A sequence-to-sequence model pre-trained for document denoising.
  • Pegasus: A model designed for abstractive summarization using a gap-sentence generation objective.
  • mT5: A multilingual variant of T5.

Models were fine-tuned on PoemSum using an 80/10/10 split and evaluated using ROUGE scores (n-gram overlap) and BERTScore (semantic similarity).

Key Results and Insights

The T5 model emerged as the top performer across all metrics, while mT5 lagged significantly behind, likely due to a "news bias" in its pre-training that caused it to hallucinate factual details not present in the poems.

Benchmark performance of T5, Pegasus, Bart, and mT5 on the PoemSum dataset.

Qualitative analysis revealed several persistent issues in AI-generated poetic summaries:

  • The "Literal Trap": Models frequently fail to distinguish between metaphors and facts (e.g., interpreting a "bleeding heart" as a physical injury).
  • Extractive Laziness: Even high-performing models often copy exact lines from the poem rather than generating a truly abstractive interpretation.
  • Contextual Blindness: Models struggle to see how individual stanzas contribute to an overarching theme, often missing deeper historical or emotional contexts.

Conclusion and Future Work

While we have mastered literal summarization, the literary frontier remains a challenge. The authors identify several areas for future growth, including expanding the dataset size (currently 3,011 samples compared to the 226k in news-based datasets like X-Sum), incorporating non-English poetry, and testing the metaphorical "intuition" of massive Large Language Models like GPT-4. PoemSum serves as a vital first step toward teaching machines to understand the "soul" of human expression.

Summary updated

BibTeX
@inproceedings{mahbub2023unveiling,
  author = {Ridwan Mahbub and Ifrad Khan and Samiha Anuva and Md Shahriar and Md. Tahmid Rahman Laskar and Sabbir Ahmed},
  title = {{Unveiling the Essence of Poetry: Introducing a Comprehensive Dataset and Benchmark for Poem Summarization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.920},
}
Figure