TL;DR
This paper explores using Retrieval-Augmented Generation (RAG) to automatically generate poetry in the distinct style of lesser-known Modern Greek interwar poets. They found that RAG-assisted models show promise in enhancing poetry generation, particularly in style and theme relevance, though a contrastive RAG approach didn't yield better results than a basic RAG system.
What Problem Does This Paper Address?
Large Language Models (LLMs) have made impressive strides in text generation, even producing poetry that can be indistinguishable from human work for non-experts. However, LLMs often struggle with languages and styles that are not well-represented in their training data. This paper tackles this challenge by focusing on a "low-resource" domain: Modern Greek poetry from the interwar period, characterized by specific stylistic features like melancholy and existential anxiety. The authors aim to improve LLM performance in generating poetry within this niche style by employing and comparing two advanced techniques: Retrieval-Augmented Generation (RAG) and contrastive learning.
Methods
The core of the paper lies in comparing three approaches for poetry generation:
- Base LLM: This is the standard LLM generation without any augmentation.
- RAG: This approach augments the LLM by retrieving relevant poems from a corpus of interwar Greek poetry based on a given theme and poet. These retrieved poems are used to construct a prompt that guides the LLM. A multilingual model and FAISS vector store are used for similarity search and embedding.
- Contrastive RAG: This method builds upon the RAG system by retrieving two sets of examples: positive examples (poems by the target poet matching the theme) and contrastive examples (poems by other poets of the same school but with different stylistic nuances). The goal is to explicitly guide the LLM to adopt the target poet's style while avoiding features present in the contrasting examples.
The system uses a dual retrieval mechanism that first finds thematically relevant poems and then splits them into positive and contrastive sets. The retrieved poems are then used to create an enhanced prompt for the LLM.
Datasets
The study utilizes a custom dataset comprising over 600 poems in text format by a group of Modern Greek interwar poets. This corpus includes works by poets such as Tellos Agras, Fotos Giofyllis, Romos Filyras, Kostas Karyotakis, Napoleon Lapathiotis, Kostas Ouranis, Mitsos Papanikolaou, and Maria Polydouri. This dataset is described as a "slightly modified and richer corpus" created by the second author and undergraduate students.
Key Insights and Ideas
- RAG's Potential for Style: The results suggest that RAG can indeed enhance poetry generation, particularly in capturing the target poet's style and relevance to the theme.
- Contrastive Learning Nuances: The contrastive RAG approach, while theoretically sound, did not outperform the basic RAG system. The authors hypothesize this might be due to the contrastive examples not being sufficiently distinctive, as they were drawn from poets within the same "poetic school."
- LLM Consistency: Base LLM models demonstrated remarkable consistency across various evaluation parameters, suggesting a robust baseline performance.
- Versification Challenge: All augmented approaches, including RAG, showed a decline in versification compared to the base LLM. This is attributed to the base LLMs potentially being trained on simpler, more rhythmic content like song lyrics.
- Metric vs. Human Evaluation: While objective metrics like Vocabulary Density (VD) and Average Words per Sentence (AWpS) showed interesting trends (e.g., increased VD in all models, possibly due to training on more analytical discourse), expert evaluations provided a more nuanced picture of stylistic adherence and creativity.
Key Results
Expert evaluators assessed generated poems based on style, theme relevance, creativity, and versification.
- GPT-40: The RAG model scored highest for style and theme.
- GPT-4 Turbo: The RAG model tied with the base LLM for theme relevance and was marginally better in style. However, the base LLM performed better in versification and creativity.
- Contrastive RAG: This model consistently showed the worst overall scores across both GPT-40 and GPT-4 Turbo evaluations.
- Objective Metrics: All models tended to increase Vocabulary Density (VD) and Average Words per Sentence (AWpS) compared to the original corpus. The RAG model showed the most increased VD but remained closer to the original AWpS.
Limitations and Future Work
The authors acknowledge three main limitations:
- Exploring RAG Variations: Further exploration of different RAG and contrastive RAG model variations is needed to fully understand their effectiveness.
- Cross-Lingual and Cross-Style Generalization: The study's findings might not directly translate to other languages or poetic styles, requiring further investigation.
- Limited Expert Pool: The evaluation relied on a small group of experts in interwar Modern Greek poetry, which could limit the generalizability of the findings.
Future work could involve:
- Developing RAG systems that are more sensitive to rhyme and rhythm, not just thematic similarity.
- Experimenting with more starkly contrasting poetic styles for contrastive training before moving to more nuanced differences within a school.
- Expanding the research to other languages and poetic styles.
Summary updated