Abstract
World models are progressively being employed across diverse fields, extending from basic environment simulation to complex scenario construction. However, existing models are mainly trained on domain-specific states and actions, and confined to single-modality state representations. In this paper, We introduce WorldGPT, a generalist world model built upon Multimodal Large Language Model (MLLM). WorldGPT acquires an understanding of world dynamics through analyzing millions of videos across various domains. To further enhance WorldGPT's capability in specialized scenarios and long-term tasks, we have integrated it with a novel cognitive architecture that combines memory offloading, knowledge retrieval, and context reflection. As for evaluation, we build WorldNet, a multimodal state transition prediction benchmark encompassing varied real-life scenarios. Conducting evaluations on WorldNet directly demonstrates WorldGPT's capability to accurately model state transition patterns, affirming its effectiveness in understanding and predicting the dynamics of complex scenarios. We further explore WorldGPT's emerging potential in serving as a world simulator, helping multimodal agents generalize to unfamiliar domains through efficiently synthesising multimodal instruction instances which are proved to be as reliable as authentic data for fine-tuning purposes.
Introduction
While Large Language Models (LLMs) excel at reasoning with text, they often lack a "world model"—an internal representation of how physical reality works. Existing world models are typically narrow, focusing only on specific domains like autonomous driving or simple robotic simulations, and are often limited to a single modality (usually just vision).
The authors identify three main gaps in current research:
- Limited Scope: Most models ignore the interplay between different senses, such as sound and sight.
- Poor Generalization: Models struggle in unfamiliar scenarios or with long-term sequences.
- Data Scarcity: There is a lack of comprehensive, large-scale datasets that map how world states transition across multiple modalities.
Methodology
WorldGPT is built on a three-part architecture consisting of multimodal encoders (LanguageBind), a Vicuna-7B core reasoning engine, and multimodal decoders. The model is trained using a Progressive State Transition Training curriculum and enhanced with a specialized Cognitive Architecture.
Progressive State Transition Training
Instead of joint training on all data simultaneously, a curriculum approach is employed. The model first learns simple unimodal transitions (e.g., video $\rightarrow$ video), then moves to cross-modal (e.g., video $\rightarrow$ audio), and finally to complex multi-modal combinations. This "easy-to-hard" strategy is crucial for stable convergence.
Cognitive Architecture
To address long-term tasks and prevent forgetting, the ContextReflector system was introduced. It manages a working memory of past predictions and a retrieval system that pulls in "external knowledge" from similar past experiences to guide current predictions. The ContextReflector acts as a filter, extracting only the relevant dynamics from retrieved memories.
The WorldNet Dataset
The researchers introduced WorldNet, a massive benchmark for multimodal state transitions:
- WorldNet-Wild: 11 million samples from unlabelled YouTube videos, automatically annotated with dense video captioning to create "action-state" pairs.
- WorldNet-Crafted: A high-quality subset of 0.3 million samples from human-labeled datasets (e.g., Ego4D, YouCook2) for fine-tuning and evaluation.
Key Insights and Results
The Necessity of Curriculum Learning
A key finding was that "naive" training often led to non-convergence. The progressive approach proved essential for handling complex modality combinations, as shown by the stable loss curves below.
"Dream Tuning" and Efficiency
WorldGPT can act as a world simulator, generating synthetic training data for other AI agents. Agents trained on these "imagined" scenarios performed nearly as well as those trained on real-world data. Furthermore, WorldGPT is significantly faster than traditional diffusion-based simulators—up to 30 times faster for certain tasks—because it predicts transitions in a compressed feature space rather than rendering pixels from scratch.
Performance and Ablation
WorldGPT consistently outperformed existing any-to-any models like CoDi and NExT-GPT. Ablation studies confirmed that adding "Reflected Knowledge" (RK) and "Reflected Memory" (RM) via the cognitive architecture significantly boosted state transition accuracy, proving the effectiveness of the reflection mechanism.
Limitations and Future Work
Despite its strengths, WorldGPT faces challenges with audio causality due to the visual bias of internet data. Additionally, performance dips in excessively long sequences suggest a processing bottleneck. Future work will involve comparisons with other emerging world models like Sora or UniSim as they become accessible for replication.
Summary updated