NeoWorld: Neural Simulation of Explorable Virtual Worlds via Progressive 3D Unfolding

Yanpeng Zhao, Shanyan Guan, Yunbo Wang, Yanhao Ge, Wei Li, Xiaokang Yang

Introduction

NeoWorld is a novel deep learning framework designed to bridge the gap between static 2D imagery and interactive 3D environments. By employing a "progressive unfolding" strategy, the system transforms a single 2D image into an expansive, explorable virtual world. This approach allows for high-fidelity physical simulations and interactive exploration while maintaining computational efficiency, a feat previously difficult to achieve in real-time applications.

The Challenge: Efficiency vs. Immersion

Creating infinite virtual worlds from a single image is a long-standing goal in computer vision. Existing methods, such as WonderWorld or WonderJourney, typically rely on 2D extrapolation or 2.5D layered representations. While effective for simple camera movements, these methods often fail during interactive exploration. Moving an object in a 2.5D scene frequently reveals geometric distortions or "holes" because the system lacks a true understanding of the object's 3D volume. Conversely, modeling an entire infinite world in full 3D is computationally prohibitive. NeoWorld addresses this trade-off by maintaining the background in lightweight 2.5D and "unfolding" foreground objects into full 3D only when interaction is required.

Methodology

NeoWorld's architecture is inspired by the concept of selective rendering—only generating detail where it is needed. The pipeline consists of three core innovations:

1. Hybrid Object-Centric Structure

The scene is initially represented as object-centric Gaussian layers. To manage an infinite number of objects without exhausting memory, NeoWorld utilizes a compact codebook of embedding vectors to track object identities and spatial relationships.

The NeoWorld framework utilizes a hybrid structure where the scene is mapped and specific objects are extracted as 3D volumes for interaction.

2. Progressive 3D Unfolding

When a user interacts with an object, the system triggers the Amodal3R module to reconstruct the object's full 3D geometry. A two-stage coarse-to-fine alignment process then seamlessly integrates this 3D model back into the 2.5D environment, ensuring spatial consistency.

3. LLM-Driven Interaction

NeoWorld integrates a Large Language Model (LLM) to act as a simulation assistant. The LLM translates natural language commands into specific physical parameters. For instance, if a user requests an object to "move up" or "bounce like jelly," the LLM identifies the target object and assigns physical properties like Young’s modulus to a physics-based simulator.

Users can interact with the scene using natural language commands, which the system translates into physical simulations.

Evaluation and Results

The framework was evaluated using a benchmark of 28 images across seven distinct artistic styles, including photorealistic, anime, and cyber-punk. For training, the researchers utilized synthetic 3D datasets such as 3D-FUTURE and ABO.

Key findings include:

  • Superior Realism: Human studies showed a 45% preference for NeoWorld over competitors, noting better 3D consistency during viewpoint shifts.
  • Command Accuracy: NeoWorld achieved a Prompt Alignment score of 61.95, significantly higher than video diffusion models like CogVideo-I2V (8.63), which struggle with complex spatial instructions.
  • Performance: The system maintains a rendering speed of approximately 18.14 seconds per view on an NVIDIA H20 GPU, making it significantly faster than high-end diffusion models.

Limitations and Future Directions

Despite its successes, NeoWorld faces challenges related to cascaded failures; an error in initial segmentation can propagate through the 3D reconstruction and alignment phases. Additionally, slight alignment issues, such as objects appearing to float, can occur. Future work aims to utilize Multimodal LLMs for direct scene perception and to develop a more unified, end-to-end trainable architecture to mitigate these issues.

Summary updated

Abstract

We introduce NeoWorld, a deep learning framework for generating interactive 3D virtual worlds from a single input image. Inspired by the on-demand worldbuilding concept in the science fiction novel Simulacron-3 (1964), our system constructs expansive environments where only the regions actively explored by the user are rendered with high visual realism through object-centric 3D representations. Unlike previous approaches that rely on global world generation or 2D hallucination, NeoWorld models key foreground objects in full 3D, while synthesizing backgrounds and non-interacted regions in 2D to ensure efficiency. This hybrid scene structure, implemented with cutting-edge representation learning and object-to-3D techniques, enables flexible viewpoint manipulation and physically plausible scene animation, allowing users to control object appearance and dynamics using natural language commands. As users interact with the environment, the virtual world progressively unfolds with increasing 3D detail, delivering a dynamic, immersive, and visually coherent exploration experience. NeoWorld significantly outperforms existing 2D and depth-layered 2.5D methods on the WorldScore benchmark.
BibTeX
@misc{zhao2025neoworld,
  author = {Yanpeng Zhao and Shanyan Guan and Yunbo Wang and Yanhao Ge and Wei Li and Xiaokang Yang},
  title = {{NeoWorld: Neural Simulation of Explorable Virtual Worlds via Progressive 3D Unfolding}},
  year = {2025},
  eprint = {2509.24441},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.24441},
}
Figure