Abstract
Modern diffusion models excel at generating images from literal descriptions but often fail to capture the metaphorical and symbolic nuances of poetry. We introduce Weighted Prompt Manipulation (WPM), a training-free, zero-shot technique that uses Large Language Models (LLMs) to assign importance weights to specific words within a poem. By adjusting cross-attention weights and text embeddings during the denoising process, WPM enables diffusion models to better visualize subtle poetic elements. Our evaluation across models like Stable Diffusion XL and Sana shows that WPM significantly improves semantic alignment and imagery richness compared to standard prompting methods.
Introduction
While text-to-image models have reached impressive levels of realism, they exhibit a notable "bias toward concrete objects." When processing poetic text, these models tend to focus on prominent nouns (e.g., "book") while ignoring more abstract or subtle descriptors (e.g., "whisper"). This limitation makes it difficult to generate visuals that reflect the mood, symbolism, or specific intent of a poet. Existing solutions often involve computationally expensive fine-tuning or struggle to maintain compositional integrity when specific details are emphasized.
Methods: Weighted Prompt Manipulation (WPM)
The authors propose WPM as a "pluggable" workflow that requires no retraining of the underlying diffusion model. The process involves three key stages:
- LLM-Guided Weighting: An LLM (specifically GPT-4o-mini) acts as a "literary director," analyzing the poem to identify visually or emotionally significant terms and assigning them numerical weights (e.g., boosting a metaphor or suppressing a literal distraction).
- Attention Modification: These weights guide the diffusion model by modifying cross-attention weights and text embeddings. This instructs the model to allocate more "attention" to specific tokens during the iterative denoising steps.
- Zero-Shot DIY Editing: Users can manually refine these weights in real-time, allowing for a "Do It Yourself" exploration of different poetic interpretations.
Key Insights: The Concrete Bias
A central finding of this research is the measurable nature of "concrete bias." Through heatmap analysis of attention maps, the researchers demonstrated that diffusion models naturally gravitate toward tangible nouns while often overlooking contextual or action-oriented words.
By using LLMs as semantic bridges, WPM can counteract this bias. The LLM understands the importance of a word within the literary context, even if it isn't a dominant physical object, and boosts its influence on the final image without breaking the overall visual structure.
Evaluation and Results
The method was evaluated using the PoemSum (3,011 poems with summaries) and MiniPo (1,001 nursery rhymes) datasets across three state-of-the-art models: Playground V3, Stable Diffusion XL (SDXL), and Sana.
- Quantitative Performance: Using the Long-CLIP metric to measure text-image alignment, WPM consistently outperformed direct poem prompts. For the Sana model, scores rose from 0.2286 to 0.2418.
- Human Evaluation: Domain experts rated WPM-generated images significantly higher in "Meaning" (4.3 vs. 3.8) and "Action Depicted" (3.9 vs. 3.2) on a 5-point scale.
- Qualitative Improvement: Visual comparisons confirmed that WPM successfully integrated previously ignored elements, such as "smoke" or "plum," into the generated scenes once their weights were adjusted.
Limitations and Future Work
WPM's effectiveness depends on the existence of some tangible visual "hooks" within the poem. For purely abstract or conceptual poetry (e.g., describing a "void of silence" without physical anchors), the underlying diffusion model may still struggle to produce meaningful visual variation.
Future Work: The researchers aim to move beyond weighting individual words to weighting entire phrases. This would allow for more sophisticated control over complex metaphors, such as "the golden breath of autumn," treating them as single semantic units rather than a collection of separate words.
Summary updated