Storybranch - generating multimedia content from novels

Rushikesh Hiray, Venelin KovatchevNAACL 2025

TL;DR

This paper introduces Storybranch, an automated system that transforms long texts like novels into engaging multimedia content, including audiobooks, animated videos, and interactive visual novel games. It addresses the challenge of maintaining consistency and detail across lengthy narratives, offering a novel approach to experiencing literature.

What Problem Does This Paper Address?

While AI has made strides in generating short-form multimedia content (images, audio clips from brief text), creating high-quality, consistent multimedia from long texts like novels remains a significant challenge. The core difficulties lie in:

  • Maintaining Consistency: Keeping track of multiple characters, their appearances, voices, and evolving traits (like clothing or age) throughout a long narrative.
  • Handling Context: Accurately identifying and generating background elements (images, sounds) that change dynamically with the story.
  • Synchronization: Seamlessly integrating audio, visuals, and character animations to match the narrative flow.
  • Scalability: Generating this content efficiently and at scale from entire books.

Existing tools often struggle with the sheer volume of information and the need for long-range coherence required for book-length content.

Methods

Storybranch employs a modular, five-step pipeline:

  1. Information Extraction: This crucial first step uses a new open-source library, BookNLP2, to parse long texts. BookNLP2 extracts entities, their properties, events, coreferences, and identifies the current speaker. This structured data is foundational.
  2. Data Processing: The extracted information is restructured to create separate entries for characters, background elements, and the narrator. This organizes the data for downstream generation.
  3. Prompt Engineering: Both structured and unstructured data are used to craft dynamic, scene- and character-specific prompts for generative AI models. This ensures that prompts are tailored to the narrative context.
  4. Content Generation: This stage leverages various Generative AI models:
  5. Audio: Text-to-Speech (TTS) models (like XTTSv2, StyleTTS2) are used for character voices and narration.
  6. Visuals: Text-to-Image models (like Stable Diffusion, FLUX) are used for character and background images. The use of "ComfyUI" workflows is highlighted for generating consistent character visuals from multiple angles.
  7. Sound Effects: Models like AudioLDM are used for background noises.
  8. Content Synchronization: The Python library 'MoviePy' is used to combine the generated audio, images, and animations into a final output. A key innovation here is the addition of a lip synchronization mechanism using the Montreal Forced Aligner (MFA) to ensure character mouth movements match spoken dialogue.

Datasets

The paper does not explicitly list large-scale datasets used for training the entire Storybranch system. Instead, it relies on:

  • Existing NLP Benchmarks: For components of BookNLP2, they reference task-specific benchmarks (e.g., from Bamman et al., 2019).
  • Human Evaluation: The primary evaluation method involves human participants assessing the quality of generated audio and comparing Storybranch to commercial alternatives.
  • Classical Novels: The system was tested on classical novels, implying these served as the input texts for generation.

The authors note that existing benchmarks for LLMs and NLP models are not directly applicable to their task of evaluating multimedia generation from long texts, necessitating their own evaluation approach.

Key Insights and Ideas

  • BookNLP2 is a Game-Changer: The development of BookNLP2 is presented as a critical enabler, providing the structured data necessary to tackle the complexity of long texts. Its modularity and compatibility with legacy systems are key.
  • Dynamic Character Generation is Crucial: The system's ability to handle changing character attributes (like clothing) dynamically, rather than relying solely on static descriptions, is vital for realism.
  • Lip Sync Adds Significant Realism: The integration of a forced alignment-based lip-sync mechanism dramatically improves the perceived quality and realism of the animated videos.
  • Modular Pipeline for Flexibility: The five-step, modular design allows for easy customization and replacement of individual AI models, making the system adaptable to future advancements.
  • Bridging the Gap to Immersive Storytelling: Storybranch moves beyond simple text-to-audio or text-to-image by creating a cohesive, multi-modal experience that can even lead to interactive games.

Key Results

  • Audiobook Quality: When augmented with dynamic sound effects, Storybranch's audio output was rated as superior to commercial alternatives like Speechify and ElevenLabs by human evaluators.
  • Video Generation: While competing systems could only generate very short video clips, Storybranch could consistently generate longer videos from books, outperforming them in quality.
  • Visual Novel Games: The system successfully converts narrative text into interactive visual novel game scripts using the Godot engine.
  • User Preferences: User studies indicated that a combination of ChatGPT for prompt generation, StyleTTS2 for text-to-speech, and Stable-open-audio for sound effects yielded the best audio results.

Limitations and Future Work

Limitations:

  • Simple Animation Style: The current animation style is described as "simplified and cartoon-like," primarily aimed at children and second-language learners.
  • Fiction-Focused: The system currently works best with fiction books that have clear descriptions; handling non-fiction is a challenge.
  • No Large-Scale Multimedia Datasets: The authors acknowledge the lack of large-scale datasets for evaluating multimedia generation from long texts, relying heavily on human evaluation.

Future Work:

  • Advanced Animation: Exploring more sophisticated video generation models to create richer animations.
  • Comic Book Generation: Adding a new output format for comic books, leveraging existing Storybranch data.
  • Non-Fiction Support: Expanding the system to handle non-fiction texts by incorporating external knowledge bases.
  • User Interaction in Games: Developing methods to add user choices and branching narratives to the visual novel games.

Summary updated

BibTeX
@inproceedings{hiray2025storybranch,
  author = {Rushikesh Hiray and Venelin Kovatchev},
  title = {{Storybranch - generating multimedia content from novels}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure