MUSE: A Multi-agent Framework for Unconstrained Story Envisioning via Closed-Loop Cognitive Orchestration

Wenzhang Sun, Zhenyu Wang, Zhangchi Hu, Chunfeng Wang, Hao Li, Wei ChenarXiv 2026 · DOI

TL;DR

This paper introduces MUSE, a novel multi-agent framework that tackles the challenge of generating long, coherent audio-visual stories from simple text prompts. By treating storytelling as a closed-loop constraint enforcement problem, MUSE significantly improves narrative consistency and cross-modal alignment, bridging the gap between high-level intent and low-level generation.

What Problem Does This Paper Address?

Generating compelling, long-form audio-visual stories is a significant hurdle in AI. Existing methods often rely on feed-forward pipelines, leading to a "intent-execution gap" where the initial narrative intent (like character identity, spatial relationships, and temporal flow) degrades over extended sequences. This results in common failures such as characters changing appearance, scenes becoming inconsistent, and narratives losing coherence, breaking viewer immersion. The core challenge is to maintain global narrative consistency across dozens of generation steps without sacrificing the expressiveness of underlying generative models.

Methods

MUSE employs a closed-loop, multi-agent orchestration approach. Instead of a single pass, it uses an iterative plan-execute-verify-revise loop.

  • Global Memory and Executable Controls: MUSE translates narrative intent into explicit, machine-executable controls for identity, spatial composition, and temporal continuity. These are stored in a shared memory H.
  • Multi-Agent System: The system is coordinated by an omni-modal controller M, which orchestrates specialist agents for different phases:
  • Pre-production Team: Focuses on identity anchoring. It establishes persistent multimodal identity states for characters (visual and vocal anchors) based on explicit appearance and vocal trait constraints, preventing drift.
  • Production Team: Handles spatial composition. It uses dynamic routing to select generation strategies, injects coarse layouts as structural priors into visual generation, and synthesizes audio conditioned on vocal anchors.
  • Post-production Team: Manages temporal synthesis. It ensures temporal continuity by conditioning each shot on the terminal state of the previous one and applies temporal planning for camera and actor motion.
  • Verification and Revision: After each execution step, a critic agent verifies the generated output against the defined constraints. If violations are detected (e.g., identity mismatch, spatial inconsistency, temporal leakage), a targeted, bounded revision is applied. This is a key differentiator from generic self-refinement, as it operates on structured controls and typed violation signals.
  • Vocal Trait Synthesis (VTS): A novel module for generating character voices directly from semantic descriptions (age, timbre, style), enabling consistent, reference-free audio identity.

Datasets

  • Training: The VTS module was trained on a large proprietary dataset of 20,000 hours of audio, annotated with semantic conditions.
  • Evaluation:
    • ViStoryBench: Used for evaluating visual consistency, focusing on identity and style.
  • MUSEBench: A newly introduced, reference-free evaluation protocol designed for open-ended storytelling. It assesses narrative coherence, cross-modal identity consistency, and other aspects using large multimodal models (LMMs) and human judgments. MUSEBench spans 30 curated narrative prompts across five genres and varying complexity.

Key Insights and Ideas

  • Intent-Execution Gap is Key: The paper argues that failures in long-form generation stem not just from model imperfections but from a fundamental gap between abstract narrative intent and concrete execution.
  • Storytelling as Constraint Enforcement: Reframing storytelling as a continuous process of planning, executing, verifying, and revising against explicit constraints is crucial for coherence.
  • Explicit Identity Anchoring: Pre-defining character identities (visual and vocal) as stable anchors, rather than relying on implicit prompt cues, is vital for preventing identity drift.
  • Structured Feedback for Targeted Revisions: Moving beyond simple prompt retries to a system that understands typed violation signals and applies localized corrections is more effective for long-horizon generation.
  • Reference-Free Evaluation is Necessary: For open-ended storytelling, benchmarks that don't rely on ground truth references are essential for truly assessing generative capabilities.

Key Results

  • Substantial Improvement: MUSE significantly improves long-horizon narrative coherence, cross-modal identity consistency, and cinematic quality compared to baselines.
  • MUSEBench Performance: On MUSEBench, MUSE achieved better script quality, narration-visual alignment, and audio generation scores.
  • ViStoryBench Performance: MUSE demonstrated strong performance on identity-related metrics, indicating reduced character drift.
  • Human-Metric Alignment: MUSEBench scores showed good correlation with human judgments for visual and narrative dimensions, validating its effectiveness.

Limitations and Future Work

  • Character Identity in Complex Scenes: Maintaining stable character identities in crowded or occluded scenes, especially with frequent viewpoint changes, remains challenging.
  • Fantasy Creature Identity: Identity inconsistency can occur during shot transitions for fantasy creatures.
  • Expressive Speech and Emotional Dynamics: While VTS enables semantic-driven voice generation, richer control over expressive speech and emotional dynamics is needed for more natural dialogue.
  • Future Work: The authors suggest exploring more robust strategies for fantasy creatures and enhancing expressive speech capabilities.

Summary updated

BibTeX
@misc{sun2026muse,
  author = {Wenzhang Sun and Zhenyu Wang and Zhangchi Hu and Chunfeng Wang and Hao Li and Wei Chen},
  title = {{MUSE: A Multi-agent Framework for Unconstrained Story Envisioning via Closed-Loop Cognitive Orchestration}},
  year = {2026},
  eprint = {2602.03028},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.03028},
}
Figure