Abstract
The current state-of-the-art video generative models can produce commercial-grade videos with highly realistic details. However, they still struggle to coherently present multiple sequential events in the stories specified by the prompts, which is foreseeable an essential capability for future long video generation scenarios. For example, top T2V generative models still fail to generate a video of the short simple story 'how to put an elephant into a refrigerator.' While existing detail-oriented benchmarks primarily focus on fine-grained metrics like aesthetic quality and spatial-temporal consistency, they fall short of evaluating models' abilities to handle event-level story presentation. To address this gap, we introduce StoryEval, a story-oriented benchmark specifically designed to assess text-to-video (T2V) models' story-completion capabilities. StoryEval features 423 prompts spanning 7 classes, each representing short stories composed of 2-4 consecutive events. We employ advanced vision-language models, such as GPT-4V and LLaVA-OV-Chat-72B, to verify the completion of each event in the generated videos, applying a unanimous voting method to enhance reliability. Our methods ensure high alignment with human evaluations, and the evaluation of 11 models reveals its challenge, with none exceeding an average story-completion rate of 50%. StoryEval provides a new benchmark for advancing T2V models and highlights the challenges and opportunities in developing next-generation solutions for coherent story-driven video generation.
Introduction: The Narrative Gap in Video Generation
The field of video generation has seen a massive leap in "low-level" quality, including aesthetic appeal, spatial consistency, and realistic textures. However, as the field moves toward long-form video generation, a new challenge emerges: event-level coherence. Current models often excel at generating high-fidelity textures and smooth motion but fail to coherently present a sequence of events—a fundamental requirement for true "world simulators."
Existing benchmarks like VBench or EvalCrafter focus on fine-grained metrics such as background flickering or spatial consistency. They do not measure if a model can follow a logical narrative arc. If a prompt asks for a sequence of actions, current models often skip steps, merge subjects, or simply fail to depict the later stages of the story. As shown in the comparison below, models often successfully start an action (like dribbling a basketball) but fail to complete the sequence (throwing the ball).
StoryEval: A Benchmark for Story Completion
To address this gap, we introduce StoryEval, a benchmark of 423 prompts designed to test video models on their ability to execute 2–4 consecutive events (e.g., "A man opens a fridge, puts an elephant in, and closes the door"). The prompt suite spans 7 categories: Human, Animal, Object, Retrieval, Creative, Easy, and Hard. These prompts were curated through a mix of real-world video captions, human brainstorming for imaginative scenarios, and guided generation using GPT-4o.
Evaluation Methodology
We developed a robust evaluation pipeline that moves beyond simple binary checks by using advanced Vision-Language Models (VLMs). The process involves a two-step querying method where the VLM first describes the video in detail and then uses that description alongside video frames to score each individual event in the prompt. To mitigate VLM hallucinations, an event is only marked as "completed" if it passes three independent verification runs (unanimous voting). The verifier specifically checks if subjects remain consistent across events. This pipeline achieved a 0.97 Spearman correlation with human annotators.
Key Results and Analysis
The evaluation of 11 models (3 closed-source, 8 open-source) reveals a significant "narrative gap." No current model achieves an average story completion rate higher than 50%.
- Kling1.5 emerged as the leader with a 40.1% average completion rate.
- Hailuo followed at 35.1%.
- Vchitect-2.0 was the top-performing open-source model at 21.7%, notably outperforming the commercial Pika1.5 (19.4%).
Performance varies significantly across categories. Models perform better on "Retrieval" prompts, which likely resemble their training data, than on "Creative" prompts. This suggests that current models rely more on memorized patterns than on a deep physical common sense of object interaction. Furthermore, performance plummeted in the "Hard" category, with even the best models struggling to exceed 16% completion.
Conclusion
StoryEval highlights that while current T2V models are visually impressive, they are not yet capable "story presenters." The 50% completion ceiling suggests that future research must move beyond texture and motion toward logical event sequencing and narrative coherence. StoryEval provides a rigorous framework for measuring progress in this essential capability for future long video generation.
Summary updated