Abstract: We present Yan, a foundational framework for interactive video generation, covering the entire pipeline from simulation and generation to editing. Specifically, Yan comprises three core modules: AAA-level Simulation, Multi-Modal Generation, and Multi-Granularity Editing. By leveraging a highly-compressed 3D-VAE and a hierarchical autoregressive captioning method, Yan achieves real-time 1080P/60FPS interactive simulation with strong generalization across domains. Our hybrid model explicitly disentangles mechanics from rendering, enabling fine-grained content editing during interaction. Yan pushes interactive video generation beyond isolated capabilities toward a comprehensive AI-driven creation paradigm.
Introduction
While generative AI has made significant strides in static video production, Interactive Generative Video (IGV)—where the output responds dynamically to user input—remains a significant challenge. Existing models often face a "trilemma," struggling to balance high visual fidelity, temporal coherence, and real-time responsiveness. Most current state-of-the-art models are limited to low resolutions or frame rates, resulting in a sluggish user experience. Yan addresses these hurdles by providing a unified pipeline for simulation, generation, and interactive editing.
The Yan Framework
Yan is composed of three interconnected modules designed to handle the complexities of real-time interactive environments.
1. AAA-Level Simulation (Yan-Sim)
To achieve high performance, Yan-Sim utilizes a modified Stable Diffusion-based architecture. It features a highly compressed 3D-VAE (increasing spatial downsampling from 8x to 32x) and a KV-cache-based shift-window denoising process. This allows the system to generate frames at 1080P/60FPS by processing a sliding window of frames, significantly reducing latency to approximately 0.11s.
2. Multi-Modal Generation (Yan-Gen)
To maintain environmental stability over long sessions, Yan employs a hierarchical captioning system. It separates "Global Captions" (describing the static world layout) from "Local Captions" (describing immediate events). This prevents the video from "drifting" or losing coherence. The model also demonstrates strong Cross-Domain Fusion, allowing it to blend styles and mechanics from different textual and visual prompts flexibly.
3. Multi-Granularity Editing (Yan-Edit)
Yan-Edit introduces a hybrid model that disentangles mechanics simulation from visual rendering. By using a depth-driven mechanics simulator to handle physics and a separate renderer for style, users can edit content on the fly using text prompts.
Dataset and Training
The framework was trained on a massive dataset derived from the 3D game Yuan Meng Star. The dataset includes over 400 million frames of interactive video data. To ensure diversity and quality, the researchers used RL-based exploration agents (PPO) and implemented a three-layer filtering system to remove anomalies such as rendering failures or physics glitches.
Key Results and Insights
- High Performance: Yan-Sim achieves a milestone of 1080P resolution at 60 FPS, outperforming previous models like GameNGen in both resolution and responsiveness.
- Disentanglement for Creativity: Separating structure from style allows for "fine-grained" editing. Users can change an object's identity (e.g., a platform to a door) without breaking the underlying physics, or swap art styles (e.g., watercolor to sci-fi) mid-interaction.
- Temporal Stability: The hierarchical context acts as a "north star," keeping the environment stable even during infinite simulations.
Conclusion and Future Work
Yan represents a significant step toward fully generative, interactive digital worlds. While it achieves high-fidelity real-time performance, future work will focus on improving perfect visual consistency over multi-hour durations and reducing the high-end hardware requirements (currently requiring NVIDIA H20 clusters). Expanding the action space beyond the current training environment remains a key goal for the next generation of creative tools.