Abstract
Text-to-image generation models often struggle with key element loss or semantic confusion in tasks involving Chinese classical poetry. Addressing this issue through fine-tuning models needs considerable training costs. Additionally, manual prompts for re-diffusion adjustments need professional knowledge. To solve this problem, we propose Poetry2Image, an iterative correction framework for images generated from Chinese classical poetry. Utilizing an external poetry dataset, Poetry2Image establishes an automated feedback and correction loop, which enhances the alignment between poetry and image through image generation models and subsequent re-diffusion modifications suggested by large language models (LLM). Using a test set of 200 sentences of Chinese classical poetry, the proposed method—when integrated with five popular image generation models—achieves an average element completeness of 70.63%, representing an improvement of 25.56% over direct image generation. In tests of semantic correctness, our method attains an average semantic consistency of 80.09%.
Introduction
Generating accurate images from Chinese classical poetry is notoriously difficult for standard text-to-image (T2I) models like DALL-E 3 or Stable Diffusion. These models often suffer from element loss (forgetting a specific flower or bird mentioned in a verse) or semantic confusion (misinterpreting metaphors or historical context). Most current T2I workflows are "open-loop," meaning they generate an image once and stop, with no mechanism to verify if the output matches the intricate details of the source text. Existing solutions, such as fine-tuning, are computationally expensive and require specialized datasets.
The Poetry2Image Framework
To address these challenges, we introduce Poetry2Image, a training-free, "closed-loop" framework that iteratively refines images through four main stages.
- Semantic Extraction: The system retrieves modern translations and expert appreciations from a poetry database. An LLM (the "Extractor") then identifies specific "key elements" (e.g., "bamboo horse," "well railing") that must appear in the image.
- Initial Generation: An initial image is produced using the poem's translation as a prompt.
- Automated Feedback Loop: An Open Vocabulary Detector (OVD) scans the generated image to identify present elements. An LLM Suggester compares these findings against the original list of key elements and issues specific editing commands: Add, Delete, Move, Replace, or Retain.
- Image Editing: The framework uses diffusion-based editing models and Segment Anything (SAM) to apply the suggested modifications. This process repeats until the image and text are semantically aligned.
Semantic Extraction and the Role of LLMs
A critical component of the framework is the use of Large Language Models (LLMs) as domain experts. Rather than just generating prompts, the LLM acts as a critic and planner. By comparing the required elements with the detected ones, it provides targeted "debugging" instructions for the image generation process.
The authors found that using modern translations for initial generation—rather than the original ancient Chinese—significantly reduces linguistic ambiguity for T2I models. In the extraction phase, Claude-3 outperformed other models like GPT-4 in accurately identifying key poetic elements.
Key Results and Performance
The framework was evaluated using a dataset of 200 well-known Chinese classical poetry sentences and integrated with five major T2I models: DALL-E 3, CogView3, Midjourney, Wenxin Yige, and Stable Diffusion.
- Improved Completeness: Poetry2Image achieved an average element completeness of 70.63%, a 25.56% improvement over direct generation. When paired with DALL-E 3, completeness reached 90.20%.
- Semantic Consistency: The method attained an average semantic consistency score of 80.09%, ensuring the core meaning of the poem was preserved.
- Complexity Threshold: The framework performs optimally on poems with 3 to 6 key elements. Beyond this, images can become aesthetically cluttered during the correction process.
Conclusion and Future Directions
Poetry2Image offers a robust, non-fine-tuning method to enhance the cultural and semantic accuracy of AI-generated imagery. However, challenges remain. The framework struggles with highly abstract or didactic poems that lack concrete physical objects. Additionally, current models often fail to recognize specific historical figures or proper nouns, identifying a generic "man" instead of a specific historical likeness. Future work will focus on improving historical accuracy and maintaining aesthetic balance during the iterative addition of elements.
Summary updated