TL;DR
Raw story text is rarely optimized for text-to-image models because it lacks explicit visual descriptions. This paper introduces a pipeline that leverages Large Language Models (LLMs) to translate narrative text into highly descriptive image prompts, significantly boosting illustration quality. The authors also introduce the SceneIllustrations dataset and demonstrate that LLMs can generate effective, atomic criteria for automated image evaluation.
What Problem Does This Paper Address?
Story visualization—generating a sequence of images to illustrate a text narrative—is a difficult task. While previous research heavily focused on maintaining visual consistency across multiple images (e.g., keeping a character looking identical from frame to frame), this paper tackles the under-explored problem of text-image alignment at the individual scene level.
Raw story text often lacks the granular visual details (such as lighting, framing, background elements, and specific physical attributes) that text-to-image models need to produce high-quality outputs. The authors investigate whether LLMs can bridge this gap by acting as an intermediary interface that infers and verbalizes implicit visual details from narrative context.
Methods
The authors propose a modular, three-step scene illustration pipeline:
- Story Fragmentation: An LLM (Claude 3.5 Sonnet) segments a story into distinct "fragments," where each fragment represents a single scene suitable for illustration.
- Scene Description (Scene Captioner): The LLM takes a fragment along with its surrounding story context and expands it into a highly detailed visual prompt (the
CAPTION). - Image Generation: The generated caption is fed into a state-of-the-art text-to-image model (such as Midjourney or FLUX) to produce the final illustration.
To evaluate the pipeline's efficacy, the authors compared the LLM-generated CAPTION prompts against three raw-text baselines:
NC-FRAGMENT: The fragment isolated from its context.VC-FRAGMENT: The fragment with the full story text dumped in as an instruction.SC-FRAGMENT: The fragment rewritten by an LLM to make contextual references explicit but concise.
Additionally, they developed a Criteria-based Evaluation method to automate quality assessment. An LLM acts as a "criteria writer" to generate a list of atomic, flexible visual requirements from the story text. A Vision-Language Model (VLM) then acts as a "criterial rater," checking the generated images against these checklist items to automatically score illustration quality.
Datasets
- ROCStories: The source corpus used for the experiments, consisting of 5-sentence, logically linked everyday stories.
- SCENEILLUSTRATIONS: The newly introduced dataset containing 2,990 items across two phases. Each item includes a story fragment, its context, two alternative AI-generated illustrations, and human pairwise quality judgments collected via Prolific.
Key Insights and Ideas
- LLMs possess implicit visual common sense: Without any visual inputs, text-only LLMs can successfully deduce what a scene should look like based purely on narrative context, translating abstract plot points into concrete visual elements (e.g., inferring facial expressions, lighting, or background objects).
- Context injection isn't enough: Simply feeding raw story context directly into a text-to-image model (
VC-FRAGMENT) yields inferior results compared to having an LLM explicitly synthesize that context into a descriptive caption. - Deconstructing evaluation into atomic criteria works: Instead of asking a VLM to give a holistic quality score (which usually aligns poorly with human judgment), breaking the evaluation down into a text-generated checklist of atomic visual traits allows VLMs to accurately predict human preferences.
Key Results
- Prompting Superiority: In human evaluations, illustrations generated using the LLM
CAPTIONprompts achieved massive win rates over the raw-text baselines: 78.1% againstNC-FRAGMENT, 74.7% againstVC-FRAGMENT, and 72.5% againstSC-FRAGMENT. - Model Comparisons: Claude 3.5 Sonnet proved to be the most effective scene captioner, yielding significantly higher win rates than Llama 3.1 and slightly outperforming GPT-4o. Among image generators, Ideogram 2.0 achieved the highest win rates in Phase 2.
- Automated Evaluation: The criterial rating approach achieved an average accuracy of ~70% in predicting human pairwise preferences, significantly outperforming the baseline VLM rater, which scored only 59%.
Limitations and Future Work
- Proprietary Model Dependency: The pipeline heavily relies on closed-weight, proprietary models (like Claude 3.5, GPT-4o, and Midjourney), which limits accessibility and reproducibility.
- Simple Story Structures: The evaluation was restricted to the ROCStories corpus, which features highly constrained, simple everyday narratives. It remains unproven how well this pipeline scales to complex, metaphor-heavy, or naturally authored literature.
- Lack of Sequential Consistency: This work explicitly isolates scene-level text-image alignment. Future work will need to integrate these LLM prompting techniques with image-to-image consistency methods to generate cohesive, multi-frame storybooks.
Summary updated