EduPo: Progress and Challenges of Automated Analysis and Generation of Czech Poetry

Rudolf Rosa, David Mareček, Tomáš Musil, Michal Chudoba, Jakub LandsperskýNAACL 2025

TL;DR

This paper introduces EduPo, a framework for automated analysis and generation of Czech poetry. It addresses the unique challenges posed by the Czech language and its poetic tradition, adapting existing NLP techniques and developing new components to achieve state-of-the-art results in both analyzing and generating Czech verse.

What Problem Does This Paper Address?

Automated analysis and generation of poetry is a challenging NLP task due to poetry's unique formal properties (meter, rhyme, syllable structure) intertwined with semantic content. While large language models (LLMs) excel at many NLP tasks, their effectiveness for poetry, especially in less-resourced languages like Czech, is limited. This is partly due to tokenization issues where poetic units like syllables don't align with LLM subwords, and LLMs often prioritize meaning over form. The paper aims to bridge this gap by developing a comprehensive framework for Czech poetry, building upon existing resources while addressing their limitations.

Methods

The EduPo framework integrates several components for both analysis and generation:

  • Analysis:
    • Phonetic Transcription: Utilizes a rule-based approach adapted from Květa, with improvements for diphthongs and vocalic consonants.
  • Syllable Counting: Focuses on counting vowels (including diphthongs and vocalic consonants) as a proxy for syllable count.
  • Rhyme Identification: Employs a rule-based reduplicant marking system.
  • Metre Detection: Leverages Květa's rule-based approach, scoring verse compatibility with potential metres.
  • Motive Identification: Uses a large language model (GPT-4o-mini) to identify key themes, a departure from previous less successful methods.
  • Stylometric Analysis: Employs sentence embeddings (SBERT) and KNN to attribute authorship and estimate author styles.
  • Generation:
    • Fine-tuning LLMs: Adapts pre-trained LLMs (Czech GPT-2 and Llama-3.1) by fine-tuning them on annotated Czech poetry data.
  • Data Formatting: Enriches training data with explicit versological annotations (rhyme scheme, metre, syllables, reduplicants) to guide the LLMs.
  • Tokenization Strategies: Explores different tokenization methods (original, BPE, syllable-based, character-based) to assess their impact on generation quality.

Datasets

  • Corpus of Czech Verse (KČV): A large, freely available corpus of 80,229 Czech poems, annotated with metadata, versological features, phonetics, and morphology. This corpus is used for both training and evaluation.
  • Czech 'Phon-Corp' lexicon: Used for phonological features and syllable boundaries.

Key Insights and Ideas

  • LLMs struggle with form: While LLMs are powerful for semantic tasks, they often prioritize meaning over formal poetic constraints, leading to nonsensical text when trying to adhere to strict rules.
  • Syllable-based tokenization is promising but challenging: Syllable-based tokenization shows potential for poetry generation but requires robust syllable splitting tools, which are lacking for Czech.
  • Explicit annotations are crucial: Enriching training data with explicit versological annotations significantly improves the LLMs' ability to adhere to formal properties like rhyme and metre.
  • Motive identification benefits from LLMs: Large language models, when prompted appropriately, can effectively identify poetic themes, outperforming previous methods.
  • Stylometry shows potential for author imitation: Stylometric analysis can achieve a notable accuracy (74% on a subset) in attributing authorship, indicating potential for imitating author styles.

Key Results

  • Rhyme Accuracy: Achieved up to 86.9% rhyme accuracy with proper data formatting.
  • Syllable Accuracy: Achieved up to 94.4% syllable accuracy with syllable-based tokenization.
  • Metre Accuracy: Achieved up to 94.8% metre accuracy with BPE tokenization.
  • Motive Identification: 76% of identified motives were marked as correct by expert annotators, with 32% of poems having all 5 motives marked as correct.
  • Style Accuracy: Achieved 28% style accuracy in imitating authors, significantly higher than random chance (8%).
  • Comparison to Base LLMs: Fine-tuned models (especially Llama-3.1) produced poems with better formal properties and meaning compared to base LLMs.

Limitations and Future Work

  • Manual Evaluation Challenges: Proper manual evaluation of poem meaningfulness is difficult and was only performed preliminarily due to issues with defining criteria and explaining them to annotators.
  • Czech-Specific Focus: The work is limited to Czech language and poetry; applicability to other languages is not evaluated.
  • Computational Resources: The size of models that can be trained is limited by available computational power.
  • Future Work:
    • Develop more robust syllable splitting tools for Czech.
  • Explore tokenizer-free approaches and Czech-specific tokenization.
  • Investigate larger base models for improved generation quality.
  • Conduct more thorough evaluations, including comparisons with commercial models.
  • Develop automated methods for evaluating poetic qualities like meaningfulness and poeticity.
  • Incorporate contemporary poetry into the training data.
  • Explore identification of poetic forms and figures of speech.

Summary updated

BibTeX
@inproceedings{rosa2025edupo,
  author = {Rudolf Rosa and David Mareček and Tomáš Musil and Michal Chudoba and Jakub Landsperský},
  title = {{EduPo: Progress and Challenges of Automated Analysis and Generation of Czech Poetry}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure