World2Minecraft: Occupancy-Driven Simulated Scenes Construction

Lechao Zhang, Haoran Xu, Jingyu Gong, Xuhong Wang, Yuan Xie, Xin Tan

Introduction

World2Minecraft is a novel framework designed to transform real-world scenes into structured, editable Minecraft environments using 3D semantic occupancy prediction. By bridging the gap between static real-world scans and interactive simulations, this approach provides a scalable way to train embodied AI agents for complex tasks like Vision-Language Navigation (VLN). To support this, the authors introduce MinecraftOcc—a massive dataset of over 100,000 high-resolution images—narrowing the reality gap and addressing the severe data scarcity in current 3D occupancy prediction models.

The Embodied AI "Trilemma"

Embodied intelligence—training agents to perceive and interact with the physical world—relies heavily on high-fidelity simulators. However, researchers currently face a trilemma of limitations:

  1. Static Scans: Platforms utilizing real-world scans (like Habitat) offer visual realism but are "frozen" and uneditable, preventing agents from physically manipulating their environment.
  2. Implicit Representations: Techniques such as Neural Radiance Fields (NeRF) or Gaussian Splatting provide stunning photorealism but lack the discrete physical properties required for an agent to "touch" or "move" objects.
  3. The Reality Gap: Voxel-based environments like Minecraft are infinitely editable and ideal for reinforcement learning, but "vanilla" Minecraft lacks the visual and structural complexity of real indoor environments.

The bridge between these domains is 3D semantic occupancy prediction—identifying what object occupies which 3D voxel. Unfortunately, existing occupancy datasets are small, noisy, and expensive to annotate, leading to models that generalize poorly to new scenes.

Methods

To solve these challenges, the authors propose a two-pronged solution: a robust scene reconstruction pipeline and an automated data acquisition system.

1. The World2Minecraft Pipeline

The World2Minecraft pipeline and Vision-Language Navigation tasks.

To convert a real room into a functional Minecraft map, the system executes the following steps:

  • Monocular Prediction: A model processes RGB images to predict 3D occupancy for individual views.
  • Volumetric Fusion: Multiple views are merged into a unified 3D representation using camera extrinsic parameters.
  • Density Filtering & Clustering: To eliminate noise, the system applies a 3D convolution to locate dense object centers and uses the DBSCAN algorithm to group them.
  • Shape-Aware Matching: Rather than placing random blocks, the system aligns predicted occupancy grids with a library of high-quality furniture templates to ensure geometric fidelity.

2. Automated Data Acquisition

The automated pipeline for constructing the MinecraftOcc dataset.

To overcome data scarcity, the authors built an automated pipeline directly inside Minecraft. Utilizing custom mods (WorldEdit, TMEO, and Screen with Coordinates), the system automatically captures first-person screenshots while simultaneously extracting perfect "ground truth" occupancy data from the game's map files. This innovation allows for the creation of massive, perfectly labeled datasets at a fraction of the cost of manual annotation.

Datasets

The project introduces two major datasets alongside existing benchmarks:

  • MinecraftOcc (New): A large-scale occupancy dataset featuring 100,165 images across 156 detailed indoor scenes (approximately 1,000 rooms). It includes 1,452 fine-grained classes mapped to 12 superclasses.
  • MinecraftVLN (New): A dataset for Vision-Language Navigation containing 3,801 samples. It tasks agents with "Next-View" and "Next-Action" prediction based on natural language instructions (e.g., "Proceed from your current location through the main door, and find the refrigerator in the living room").
  • NYUv2 & OccScanNet: Existing real-world datasets used for benchmarking and cross-dataset training experiments.

Key Insights and Results

Voxels as a Common Language

The core insight of the paper is that the discrete voxel structure of 3D occupancy prediction is a natural mathematical match for Minecraft’s block-based world, bypassing the need for complex mesh-to-block conversions.

Sim-to-Real-to-Sim

The authors use Minecraft to generate data that improves occupancy models, which are then used to bring real-world scenes back into Minecraft. Training on MinecraftOcc acts as powerful data augmentation. When combined with the NYUv2 training set, it improved a model's performance on real-world test data by +0.21 mIoU.

Superior Image Quality and VLN Success

MinecraftOcc significantly outperforms existing datasets in image clarity, achieving a Laplacian Variance (LV) score of 274,305 (compared to 57,369 for NYUv2). Furthermore, fine-tuning large vision-language models (like Qwen2.5-VL) on the MinecraftVLN dataset led to massive gains. For the "Next-Action" task, accuracy jumped from 0.19 (no training) to 0.72 (after SFT). The authors also successfully demonstrated a Gemini-2.5-Pro agent navigating a reconstructed scene in real-time to find specific objects based on text prompts.

Limitations and Future Work

Despite high-fidelity mods, Minecraft remains a blocky environment, leaving a residual visual "style" difference compared to photorealistic scans. Additionally, initial automated reconstructions can sometimes have "floating artifacts" or holes, requiring brief manual cleanup (though the authors note their SceneForge tool makes refining an automated reconstruction 7x faster than building from scratch). Future work aims to extend this pipeline from static indoor scenes to dynamic environments with moving objects or NPCs, and eventually scale to vast outdoor landscapes.

Summary updated

Abstract

Embodied intelligence requires high-fidelity simulation environments to support perception and decision-making, yet existing platforms often suffer from data contamination and limited flexibility. To mitigate this, we propose World2Minecraft to convert real-world scenes into structured Minecraft environments based on 3D semantic occupancy prediction. In the reconstructed scenes, we can effortlessly perform downstream tasks such as Vision-Language Navigation(VLN). However, we observe that reconstruction quality heavily depends on accurate occupancy prediction, which remains limited by data scarcity and poor generalization in existing models. We introduce a low-cost, automated, and scalable data acquisition pipeline for creating customized occupancy datasets, and demonstrate its effectiveness through MinecraftOcc, a large-scale dataset featuring 100,165 images from 156 richly detailed indoor scenes. Extensive experiments show that our dataset provides a critical complement to existing datasets and poses a significant challenge to current SOTA methods. These findings contribute to improving occupancy prediction and highlight the value of World2Minecraft in providing a customizable and editable platform for personalized embodied AI research. We will publicly release the dataset and the complete generation framework to ensure reproducibility and encourage future work.
BibTeX
@inproceedings{zhang2026world,
  author = {Lechao Zhang and Haoran Xu and Jingyu Gong and Xuhong Wang and Yuan Xie and Xin Tan},
  title = {{World2Minecraft: Occupancy-Driven Simulated Scenes Construction}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=dc90uPqxWF},
}
Figure