TL;DR
This paper introduces a novel pipeline for automatically generating illustrations for narrative scenes using Large Language Models (LLMs) and text-to-image models. They also release the SCENEILLUSTRATIONS dataset, a valuable resource for future research in cross-modal narrative transformation, and demonstrate that LLMs can effectively infer and verbalize visual scene knowledge from text.
What Problem Does This Paper Address?
The paper tackles the challenge of automatically visualizing stories that are currently only in text form. While text-to-image models are powerful, raw story text often lacks the detailed visual descriptions needed to generate optimal illustrations. The authors propose using LLMs as an intermediary to bridge this gap, transforming story text into effective prompts for image generation. This addresses the problem of "story visualization," specifically focusing on the text-image alignment aspect rather than image-image consistency across scenes.
Methods
The core of the paper is a pipeline that takes raw story text and generates scene illustrations. This pipeline involves:
- Story Fragmentation: An LLM (specifically CLAUDE-3.5) is used to segment a story into distinct "fragments," each representing a scene that can be illustrated.
- Scene Descriptions: LLMs are employed to generate descriptive captions (scene descriptions) for each fragment. The paper explores several types of scene descriptions:
- NC-FRAGMENT: The raw fragment text itself.
- VC-FRAGMENT: The full story context inserted into the description.
- SC-FRAGMENT: A rewritten version of the fragment that explicitly incorporates context.
- CAPTION: A descriptive caption generated by an LLM specifically for illustration.
- Image Generation: These scene descriptions are then fed as prompts to text-to-image models (e.g., Midjourney v6.1, FLUX-1.1-PRO) to produce the final illustrations.
The authors also investigate the use of LLMs for evaluating illustration quality by generating "criteria sets" that describe desired visual characteristics and then using visually-enabled LLMs to rate illustrations against these criteria.
Datasets
- ROCStories: The authors used the ROCStories corpus for generating illustrations. This corpus is known for its tightly structured, five-sentence stories that depict causally linked events, making them suitable for scene-based visualization.
- SCENEILLUSTRATIONS Dataset: This is a newly introduced dataset created by the authors. It consists of 2990 items, each containing a story fragment, its context, and two illustrations depicting the fragment. These illustrations vary based on the scene description type and/or the image generator used. The dataset is annotated with human quality judgments.
Key Insights and Ideas
- LLMs as Effective Prompt Engineers for Images: The research strongly suggests that LLMs are not just capable of understanding text but can also translate that understanding into effective visual prompts for image generation models. The "CAPTION" scene descriptions, generated by LLMs, consistently led to higher quality illustrations compared to using raw text or context-enhanced text.
- Implicit Visual Knowledge in Text: LLMs can infer and verbalize visual characteristics of scenes that are only implicitly conveyed by the story text, without needing direct visual input. This is crucial for generating rich and accurate illustrations.
- LLMs as Quality Assessors: The study demonstrates that LLMs can be used to generate evaluation criteria for image quality and even act as "criterial raters" to assess illustrations, outperforming baseline rating methods. This opens avenues for automated evaluation in image generation tasks.
- The Importance of Scene Description Type: The type of scene description fed to the image generator significantly impacts illustration quality. LLM-generated descriptive captions ("CAPTION") were demonstrably superior to simpler text-based descriptions.
Key Results
- Illustration Quality Improvement: Illustrations generated using LLM-crafted "CAPTION" scene descriptions significantly outperformed those generated from baseline descriptions (NC-FRAGMENT, VC-FRAGMENT, SC-FRAGMENT), with win rates around 73-78% in Phase 1.
- Inter-Annotator Agreement: Human annotators showed moderate agreement (κu=0.436 in Phase 1) when judging illustration quality, with agreement being higher when illustrations used different scene descriptions, highlighting the impact of the description.
- LLM Rater Performance: In predicting human judgments of illustration quality, criterial raters (LLMs using generated criteria) achieved significantly higher accuracy (around 70%) compared to baseline raters (around 59%).
- LLM Captioner Comparison: In Phase 2, CLAUDE-3.5 generally outperformed GPT-4O and LLAMA-3.1 in generating descriptive captions that led to better illustrations.
Limitations and Future Work
- Proprietary Models: The pipeline relies heavily on proprietary, closed-weight models (LLMs and image generators), limiting accessibility and reproducibility. The authors note that while the gap between open and closed-weight models is narrowing, many capable models remain proprietary.
- Prompt Design: Optimizing prompts for LLMs and text-to-image models is still an iterative, trial-and-error process. The authors used a principled approach rather than quantitative optimization, leaving room for further prompt tuning.
- Story Corpus: The use of the ROCStories corpus, while suitable for controlled experiments, might not generalize to more complex, "naturally" authored narratives.
- Future Work: The authors plan to extend their work to generate multi-scene image sequences that depict complete stories, addressing image-image alignment challenges (ensuring visual consistency across scenes) in addition to text-image alignment. They also aim to explore more advanced approaches for prompt optimization and investigate the performance of different image generators and LLMs more thoroughly.
Summary updated