Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generation

Yiping Wang, Xuehai He, Kuan Wang, Luyao Ma, Jianwei Yang, Shuohang Wang, Simon Shaolei Du, Yelong Shen

Abstract

The current state-of-the-art video generative models can produce commercial-grade videos with highly realistic details. However, they still struggle to coherently present multiple sequential events in specific short stories, which is foreseeable an essential capability for future long video generation scenarios. For example, top T2V generative models still fail to generate a video of the short simple story "how to put an elephant into a refrigerator." While existing detail-oriented benchmarks primarily focus on fine-grained metrics like aesthetic quality and spatial-temporal consistency, they fall short of evaluating models' abilities to handle event-level story presentation. To address this gap, we introduce StoryEval, a story-oriented benchmark specifically designed to assess text-to-video (T2V) models' story-completion capabilities. StoryEval features 423 prompts spanning 7 classes, each representing short stories composed of 2-4 consecutive events. We employ Vision-Language Models, such as GPT-4o and LLaVA-OV-Chat-72B, to verify the completion of each event in the generated videos, applying a unanimous voting method to enhance reliability. Our methods ensure high alignment with human evaluations, and the evaluation of 11 models reveals its challenge, with none exceeding an average story-completion rate of 50%. StoryEval provides a new benchmark for advancing T2V models and highlights the challenges and opportunities in developing next-generation solutions for coherent story-driven video generation.

Introduction: Beyond Visual Realism

We are currently in the "Sora era" of video generation, where models are often marketed as "world simulators." However, most existing benchmarks (such as VBench or EvalCrafter) evaluate videos based on "low-level" features: lighting consistency, motion blur, and subject persistence. While these are important for aesthetic quality, they miss a critical requirement for long-form video: consecutive event logic.

Figure 6: Examples of the 'elephant in the refrigerator' challenge across different model outputs, illustrating the difficulty of sequential event logic.

If you ask a model to "open a fridge, put an elephant inside, and close the door," it might generate a beautiful kitchen, but it almost certainly won't show all three actions in a coherent sequence. StoryEval addresses this gap by shifting the focus from aesthetics to "high-level" event consistency, measuring a model's "completion rate"—the percentage of discrete, sequential events in a prompt that actually appear in the generated video.

The StoryEval Benchmark

The researchers introduced the StoryEval Prompt Suite, consisting of 423 prompts across seven categories: Human, Animal, Object, Retrieval, Creative, Easy, and Hard. These prompts are specifically designed to be "short stories" containing 2–4 consecutive events that should fit within a 10-second video. Prompts were curated via three methods:

  • Retrieval: Captions from real-world videos.
  • Human Brainstorming: Imaginative but plausible sequences.
  • GPT-4o Guided Generation: Using LLMs to expand the diversity of actions and subjects.

Methodology: VLMs as Judges

Evaluating "storytelling" is inherently subjective and expensive if done by humans. To scale this, the researchers developed a robust automated pipeline using Vision-Language Models (VLMs) as judges.

Figure 4: Basketball sequences used to evaluate referent consistency and event completion across different prompt categories.

To ensure reliability, the pipeline employs several key strategies:

  1. Two-Step Querying: Instead of asking a VLM a binary "Did this happen?", they first ask the VLM to describe the video in detail. This description is then provided back to the VLM to score the completion of specific events, a "chain-of-thought" approach that stabilizes results.
  2. Unanimous Voting: To combat VLM hallucinations, the authors use a "unanimous voting" strategy ($N$-over-$N$). For GPT-4o, an event is only marked as "completed" if the model confirms it in three separate runs.
  3. Referent Consistency: Judges are instructed to check if the subject remains the same. If a man dribbles a ball in frame one, but a different man throws it later, the event is marked as a failure.

Key Insights and Results

The evaluation of 11 leading models revealed a stark reality: no model exceeded a 50% average completion rate. Even the most advanced models are essentially failing half of their storytelling assignments.

  • The Performance Hierarchy: Kling-1.5 emerged as the leader with a 40.1% average completion rate, followed by Hailuo at 35.1%. Among open-source models, Vchitect-2.0 was the surprise winner (21.7%), outperforming Pika-1.5.
  • The "Creative" Gap: Closed-source models have a massive lead in "Creative" tasks—scenarios that don't exist in the real world—suggesting proprietary models possess better internal world models or reasoning capabilities.
  • Quality vs. Logic Trade-off: Interestingly, some models that rank high on traditional temporal consistency benchmarks performed poorly on StoryEval. This suggests that models might "freeze" or minimize motion to maintain visual quality, effectively sacrificing the story to look "pretty."

Conclusion and Future Work

StoryEval highlights that complex causal chains remain a major hurdle. Models often show the "start" and "end" of a story but skip the "middle" interaction. Future work must address VLM blind spots regarding fast movement and safety filters, while also scaling these benchmarks as video generation moves toward longer, minute-plus durations. Developing architectural priors for physical interaction and causality will be essential for the next generation of world simulators.

Summary updated

Abstract

The current state-of-the-art video generative models can produce commercial-grade videos with highly realistic details. However, they still struggle to coherently present multiple sequential events in specific short stories, which is foreseeable an essential capability for future long video generation scenarios. For example, top T2V generative models still fail to generate a video of the short simple story "how to put an elephant into a refrigerator." While existing detail-oriented benchmarks primarily focus on fine-grained metrics like aesthetic quality and spatial-temporal consistency, they fall short of evaluating models' abilities to handle event-level story presentation. To address this gap, we introduce StoryEval, a story-oriented benchmark specifically designed to assess text-to-video (T2V) models' story-completion capabilities. StoryEval features 423 prompts spanning 7 classes, each representing short stories composed of 2-4 consecutive events. We employ Vision-Language Models, such as GPT-4o and LLaVA-OV-Chat-72B, to verify the completion of each event in the generated videos, applying a unanimous voting method to enhance reliability. Our methods ensure high alignment with human evaluations, and the evaluation of 11 models reveals its challenge, with none exceeding an average story-completion rate of 50%. StoryEval provides a new benchmark for advancing T2V models and highlights the challenges and opportunities in developing next-generation solutions for coherent story-driven video generation. Project website is available at https://ypwang61.github.io/project/StoryEval.
BibTeX
@inproceedings{wang2025is,
  author = {Yiping Wang and Xuehai He and Kuan Wang and Luyao Ma and Jianwei Yang and Shuohang Wang and Simon Shaolei Du and Yelong Shen},
  title = {{Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=TvybrSBlH8},
}
Figure