WorldGPT: Empowering LLM as Multimodal World Model

Zhiqi Ge, Hongzhe Huang, Mingze Zhou, Juncheng Li, Guoming Wang, Siliang Tang, Yueting Zhuang

Abstract

World models are progressively being employed across diverse fields, extending from basic environment simulation to complex scenario construction. However, existing models are mainly trained on domain-specific states and actions, and confined to single-modality state representations. In this paper, We introduce WorldGPT, a generalist world model built upon Multimodal Large Language Model (MLLM). WorldGPT acquires an understanding of world dynamics through analyzing millions of videos across various domains. To further enhance WorldGPT's capability in specialized scenarios and long-term tasks, we have integrated it with a novel cognitive architecture that combines memory offloading, knowledge retrieval, and context reflection. As for evaluation, we build WorldNet, a multimodal state transition prediction benchmark encompassing varied real-life scenarios. Conducting evaluations on WorldNet directly demonstrates WorldGPT's capability to accurately model state transition patterns, affirming its effectiveness in understanding and predicting the dynamics of complex scenarios. We further explore WorldGPT's emerging potential in serving as a world simulator, helping multimodal agents generalize to unfamiliar domains through efficiently synthesising multimodal instruction instances which are proved to be as reliable as authentic data for fine-tuning purposes.

Introduction

While Large Language Models (LLMs) excel at reasoning with text, they often lack a "world model"—an internal representation of how physical reality works. Existing world models are typically narrow, focusing only on specific domains like autonomous driving or simple robotic simulations, and are often limited to a single modality (usually just vision).

The authors identify three main gaps in current research:

  1. Limited Scope: Most models ignore the interplay between different senses, such as sound and sight.
  2. Poor Generalization: Models struggle in unfamiliar scenarios or with long-term sequences.
  3. Data Scarcity: There is a lack of comprehensive, large-scale datasets that map how world states transition across multiple modalities.

Methodology

WorldGPT is built on a three-part architecture consisting of multimodal encoders (LanguageBind), a Vicuna-7B core reasoning engine, and multimodal decoders. The model is trained using a Progressive State Transition Training curriculum and enhanced with a specialized Cognitive Architecture.

The WorldGPT training pipeline includes a four-step progressive pretraining curriculum followed by cognitive-augmented tuning.

Progressive State Transition Training

Instead of joint training on all data simultaneously, a curriculum approach is employed. The model first learns simple unimodal transitions (e.g., video $\rightarrow$ video), then moves to cross-modal (e.g., video $\rightarrow$ audio), and finally to complex multi-modal combinations. This "easy-to-hard" strategy is crucial for stable convergence.

Cognitive Architecture

To address long-term tasks and prevent forgetting, the ContextReflector system was introduced. It manages a working memory of past predictions and a retrieval system that pulls in "external knowledge" from similar past experiences to guide current predictions. The ContextReflector acts as a filter, extracting only the relevant dynamics from retrieved memories.

The WorldNet Dataset

The researchers introduced WorldNet, a massive benchmark for multimodal state transitions:

  • WorldNet-Wild: 11 million samples from unlabelled YouTube videos, automatically annotated with dense video captioning to create "action-state" pairs.
  • WorldNet-Crafted: A high-quality subset of 0.3 million samples from human-labeled datasets (e.g., Ego4D, YouCook2) for fine-tuning and evaluation.

Key Insights and Results

The Necessity of Curriculum Learning

A key finding was that "naive" training often led to non-convergence. The progressive approach proved essential for handling complex modality combinations, as shown by the stable loss curves below.

Loss curves demonstrating the convergence behavior of different training strategies, highlighting the stability of the progressive approach.

"Dream Tuning" and Efficiency

WorldGPT can act as a world simulator, generating synthetic training data for other AI agents. Agents trained on these "imagined" scenarios performed nearly as well as those trained on real-world data. Furthermore, WorldGPT is significantly faster than traditional diffusion-based simulators—up to 30 times faster for certain tasks—because it predicts transitions in a compressed feature space rather than rendering pixels from scratch.

Performance and Ablation

WorldGPT consistently outperformed existing any-to-any models like CoDi and NExT-GPT. Ablation studies confirmed that adding "Reflected Knowledge" (RK) and "Reflected Memory" (RM) via the cognitive architecture significantly boosted state transition accuracy, proving the effectiveness of the reflection mechanism.

Limitations and Future Work

Despite its strengths, WorldGPT faces challenges with audio causality due to the visual bias of internet data. Additionally, performance dips in excessively long sequences suggest a processing bottleneck. Future work will involve comparisons with other emerging world models like Sora or UniSim as they become accessible for replication.

Summary updated

Abstract

World models are progressively being employed across diverse fields, extending from basic environment simulation to complex scenario construction. However, existing models are mainly trained on domain-specific states and actions, and confined to single-modality state representations. In this paper, We introduce WorldGPT, a generalist world model built upon Multimodal Large Language Model (MLLM). WorldGPT acquires an understanding of world dynamics through analyzing millions of videos across various domains. To further enhance WorldGPT's capability in specialized scenarios and long-term tasks, we have integrated it with a novel cognitive architecture that combines memory offloading, knowledge retrieval, and context reflection. As for evaluation, we build WorldNet, a multimodal state transition prediction benchmark encompassing varied real-life scenarios. Conducting evaluations on WorldNet directly demonstrates WorldGPT's capability to accurately model state transition patterns, affirming its effectiveness in understanding and predicting the dynamics of complex scenarios. We further explore WorldGPT's emerging potential in serving as a world simulator, helping multimodal agents generalize to unfamiliar domains through efficiently synthesising multimodal instruction instances which are proved to be as reliable as authentic data for fine-tuning purposes. The project is available on \url{https://github.com/DCDmllm/WorldGPT}.
BibTeX
@inproceedings{ge2024worldgpt,
  author = {Zhiqi Ge and Hongzhe Huang and Mingze Zhou and Juncheng Li and Guoming Wang and Siliang Tang and Yueting Zhuang},
  title = {{WorldGPT: Empowering LLM as Multimodal World Model}},
  booktitle = {ACM Multimedia},
  year = {2024},
  doi = {10.1145/3664647.3681488},
}
Figure