Abstract
Generating logically coherent video from text (T2V) for reasoning-intensive tasks like mathematical problem-solving presents a significant challenge for Vision-Language Models (VLMs). Therefore, we introduce VisualEDU, a benchmark based on the Manim package to rigorously evaluate VLM capabilities in producing coherent, step-by-step video solutions for educational purposes. We propose a framework that integrates meta-prompt learning, visual and code feedback, and a modular drawing toolkit to enhance output quality. Novel metrics for temporal consistency, logical correctness, and visual clarity are proposed. Extensive experiments across nine VLMs reveal that while advanced proprietary models show promise, all struggle significantly with increasing task complexity (e.g., Claude-3.7-Sonnet and GPT-4o performance falls below 56% on difficult tasks), highlighting limitations in code generation, visual feedback correction, and precise tool invocation.
Introduction
While generative AI has made massive strides in text-to-video (T2V) synthesis, these models often prioritize visual fidelity over structural logic. In educational contexts, a video is only useful if the visual representation (e.g., a graph) accurately matches the underlying mathematical logic. Current T2V benchmarks focus on general content but ignore reasoning-intensive tasks. Generating a math tutorial requires a "triple threat" of capabilities: mathematical reasoning, precise code generation (using libraries like Manim), and visual comprehension to refine spatial errors like overlapping text or off-screen elements.
The VisualEDU Framework
To address these challenges, the researchers developed the VisualEDU Framework, a multi-agent pipeline designed to move beyond simple generation.
The framework consists of several key components:
- Modular Drawing Tool Library: A structured JSON-formatted compendium of valid Manim modules to prevent code hallucinations.
- Meta-Prompting: Decomposes high-level requests into manageable subtasks (e.g., "draw the axes," "plot the function").
- Visual Feedback Loop: A novel component where the VLM acts as a critic, inspecting rendered keyframes to identify layout issues and generating correction prompts to refine the code.
- Multi-Agent Architecture: Specialized agents for solving, scripting, coding, and visual evaluation.
The VisualEDU Dataset
The benchmark utilizes a dataset of 200 math problems across six domains: Functions, Plane Geometry, Analytic Geometry, Solid Geometry, Probability, and Calculus. Problems are categorized into Easy, Medium, and Hard levels based on step count and animation complexity. Each entry includes the problem, a step-by-step solution, the Manim code, and the rendered video.
Experimental Results and Analysis
The researchers evaluated nine mainstream VLMs using metrics such as Logical Pass Rate (LPR), Feedback Effectiveness (FE), and Aesthetic Score (AS).
Key findings include:
- The "Complexity Wall": As shown in the heatmap, there is a sharp performance degradation as tasks move from "Easy" to "Hard." Even top models like Claude-3.7-Sonnet and GPT-4o struggle to maintain logical and visual consistency as mathematical problems become more advanced.
- Proprietary Dominance: Claude-3.7-Sonnet led with 64.9%, followed by Gemini-2.5-Pro (62.1%) and GPT-4o (61.5%). Open-source models like Qwen2.5-VL-72B still lag significantly in Logical Pass Rate.
- Feedback Utility: Ablation studies confirmed that visual feedback loops significantly improve scores, indicating that models are often better at identifying errors in rendered frames than preventing them during initial coding.
Conclusion and Future Work
VisualEDU highlights the limitations of current VLMs in programmatic video generation for education. Future work should expand the benchmark to other scientific domains and languages. Additionally, exploring Reinforcement Learning (RL) to fine-tune models for the iterative "code-render-critique" cycle could bridge the gap in complex tool invocation and long-context reasoning.
Summary updated