LLMs Behind the Scenes: Enabling Narrative Scene Illustration

Melissa Roemmele, John Joon Young Chung, Taewook Kim, Yuqian Sun, Alex Calderwood, Max Kreminski

TL;DR

Raw story text is rarely optimized for text-to-image models because it lacks explicit visual descriptions. This paper introduces a pipeline that leverages Large Language Models (LLMs) to translate narrative text into highly descriptive image prompts, significantly boosting illustration quality. The authors also introduce the SceneIllustrations dataset and demonstrate that LLMs can generate effective, atomic criteria for automated image evaluation.

What Problem Does This Paper Address?

Story visualization—generating a sequence of images to illustrate a text narrative—is a difficult task. While previous research heavily focused on maintaining visual consistency across multiple images (e.g., keeping a character looking identical from frame to frame), this paper tackles the under-explored problem of text-image alignment at the individual scene level.

Raw story text often lacks the granular visual details (such as lighting, framing, background elements, and specific physical attributes) that text-to-image models need to produce high-quality outputs. The authors investigate whether LLMs can bridge this gap by acting as an intermediary interface that infers and verbalizes implicit visual details from narrative context.

Methods

The authors propose a modular, three-step scene illustration pipeline:

  1. Story Fragmentation: An LLM (Claude 3.5 Sonnet) segments a story into distinct "fragments," where each fragment represents a single scene suitable for illustration.
  2. Scene Description (Scene Captioner): The LLM takes a fragment along with its surrounding story context and expands it into a highly detailed visual prompt (the CAPTION).
  3. Image Generation: The generated caption is fed into a state-of-the-art text-to-image model (such as Midjourney or FLUX) to produce the final illustration.
Figure 1: Overview of scene illustration pipeline

To evaluate the pipeline's efficacy, the authors compared the LLM-generated CAPTION prompts against three raw-text baselines:

  • NC-FRAGMENT: The fragment isolated from its context.
  • VC-FRAGMENT: The fragment with the full story text dumped in as an instruction.
  • SC-FRAGMENT: The fragment rewritten by an LLM to make contextual references explicit but concise.

Additionally, they developed a Criteria-based Evaluation method to automate quality assessment. An LLM acts as a "criteria writer" to generate a list of atomic, flexible visual requirements from the story text. A Vision-Language Model (VLM) then acts as a "criterial rater," checking the generated images against these checklist items to automatically score illustration quality.

Datasets

  • ROCStories: The source corpus used for the experiments, consisting of 5-sentence, logically linked everyday stories.
  • SCENEILLUSTRATIONS: The newly introduced dataset containing 2,990 items across two phases. Each item includes a story fragment, its context, two alternative AI-generated illustrations, and human pairwise quality judgments collected via Prolific.

Key Insights and Ideas

  • LLMs possess implicit visual common sense: Without any visual inputs, text-only LLMs can successfully deduce what a scene should look like based purely on narrative context, translating abstract plot points into concrete visual elements (e.g., inferring facial expressions, lighting, or background objects).
  • Context injection isn't enough: Simply feeding raw story context directly into a text-to-image model (VC-FRAGMENT) yields inferior results compared to having an LLM explicitly synthesize that context into a descriptive caption.
  • Deconstructing evaluation into atomic criteria works: Instead of asking a VLM to give a holistic quality score (which usually aligns poorly with human judgment), breaking the evaluation down into a text-generated checklist of atomic visual traits allows VLMs to accurately predict human preferences.

Key Results

  • Prompting Superiority: In human evaluations, illustrations generated using the LLM CAPTION prompts achieved massive win rates over the raw-text baselines: 78.1% against NC-FRAGMENT, 74.7% against VC-FRAGMENT, and 72.5% against SC-FRAGMENT.
  • Model Comparisons: Claude 3.5 Sonnet proved to be the most effective scene captioner, yielding significantly higher win rates than Llama 3.1 and slightly outperforming GPT-4o. Among image generators, Ideogram 2.0 achieved the highest win rates in Phase 2.
  • Automated Evaluation: The criterial rating approach achieved an average accuracy of ~70% in predicting human pairwise preferences, significantly outperforming the baseline VLM rater, which scored only 59%.

Limitations and Future Work

  • Proprietary Model Dependency: The pipeline heavily relies on closed-weight, proprietary models (like Claude 3.5, GPT-4o, and Midjourney), which limits accessibility and reproducibility.
  • Simple Story Structures: The evaluation was restricted to the ROCStories corpus, which features highly constrained, simple everyday narratives. It remains unproven how well this pipeline scales to complex, metaphor-heavy, or naturally authored literature.
  • Lack of Sequential Consistency: This work explicitly isolates scene-level text-image alignment. Future work will need to integrate these LLM prompting techniques with image-to-image consistency methods to generate cohesive, multi-frame storybooks.

Summary updated

Abstract

Generative AI has established the opportunity to readily transform content from one medium to another. This capability is especially powerful for storytelling, where visual illustrations can illuminate a story originally expressed in text. In this paper, we focus on the task of narrative scene illustration, which involves automatically generating an image depicting a scene in a story. Motivated by recent progress on text-to-image models, we consider a pipeline that uses LLMs as an interface for prompting text-to-image models to generate scene illustrations given raw story text. We apply variations of this pipeline to a prominent story corpus in order to synthesize illustrations for scenes in these stories. We conduct a human annotation task to obtain pairwise quality judgments for these illustrations. The outcome of this process is the SceneIllustrations dataset, which we release as a new resource for future work on cross-modal narrative transformation. Through our analysis of this dataset and experiments modeling illustration quality, we demonstrate that LLMs can effectively verbalize scene knowledge implicitly evoked by story text. Moreover, this capability is impactful for generating and evaluating illustrations.
BibTeX
@inproceedings{roemmele2025llms,
  author = {Melissa Roemmele and John Joon Young Chung and Taewook Kim and Yuqian Sun and Alex Calderwood and Max Kreminski},
  title = {{LLMs Behind the Scenes: Enabling Narrative Scene Illustration}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure