Introduction
World2Minecraft is a novel framework designed to transform real-world scenes into structured, editable Minecraft environments using 3D semantic occupancy prediction. By bridging the gap between static real-world scans and interactive simulations, this approach provides a scalable way to train embodied AI agents for complex tasks like Vision-Language Navigation (VLN). To support this, the authors introduce MinecraftOcc—a massive dataset of over 100,000 high-resolution images—narrowing the reality gap and addressing the severe data scarcity in current 3D occupancy prediction models.
The Embodied AI "Trilemma"
Embodied intelligence—training agents to perceive and interact with the physical world—relies heavily on high-fidelity simulators. However, researchers currently face a trilemma of limitations:
- Static Scans: Platforms utilizing real-world scans (like Habitat) offer visual realism but are "frozen" and uneditable, preventing agents from physically manipulating their environment.
- Implicit Representations: Techniques such as Neural Radiance Fields (NeRF) or Gaussian Splatting provide stunning photorealism but lack the discrete physical properties required for an agent to "touch" or "move" objects.
- The Reality Gap: Voxel-based environments like Minecraft are infinitely editable and ideal for reinforcement learning, but "vanilla" Minecraft lacks the visual and structural complexity of real indoor environments.
The bridge between these domains is 3D semantic occupancy prediction—identifying what object occupies which 3D voxel. Unfortunately, existing occupancy datasets are small, noisy, and expensive to annotate, leading to models that generalize poorly to new scenes.
Methods
To solve these challenges, the authors propose a two-pronged solution: a robust scene reconstruction pipeline and an automated data acquisition system.
1. The World2Minecraft Pipeline
To convert a real room into a functional Minecraft map, the system executes the following steps:
- Monocular Prediction: A model processes RGB images to predict 3D occupancy for individual views.
- Volumetric Fusion: Multiple views are merged into a unified 3D representation using camera extrinsic parameters.
- Density Filtering & Clustering: To eliminate noise, the system applies a 3D convolution to locate dense object centers and uses the DBSCAN algorithm to group them.
- Shape-Aware Matching: Rather than placing random blocks, the system aligns predicted occupancy grids with a library of high-quality furniture templates to ensure geometric fidelity.
2. Automated Data Acquisition
To overcome data scarcity, the authors built an automated pipeline directly inside Minecraft. Utilizing custom mods (WorldEdit, TMEO, and Screen with Coordinates), the system automatically captures first-person screenshots while simultaneously extracting perfect "ground truth" occupancy data from the game's map files. This innovation allows for the creation of massive, perfectly labeled datasets at a fraction of the cost of manual annotation.
Datasets
The project introduces two major datasets alongside existing benchmarks:
- MinecraftOcc (New): A large-scale occupancy dataset featuring 100,165 images across 156 detailed indoor scenes (approximately 1,000 rooms). It includes 1,452 fine-grained classes mapped to 12 superclasses.
- MinecraftVLN (New): A dataset for Vision-Language Navigation containing 3,801 samples. It tasks agents with "Next-View" and "Next-Action" prediction based on natural language instructions (e.g., "Proceed from your current location through the main door, and find the refrigerator in the living room").
- NYUv2 & OccScanNet: Existing real-world datasets used for benchmarking and cross-dataset training experiments.
Key Insights and Results
Voxels as a Common Language
The core insight of the paper is that the discrete voxel structure of 3D occupancy prediction is a natural mathematical match for Minecraft’s block-based world, bypassing the need for complex mesh-to-block conversions.
Sim-to-Real-to-Sim
The authors use Minecraft to generate data that improves occupancy models, which are then used to bring real-world scenes back into Minecraft. Training on MinecraftOcc acts as powerful data augmentation. When combined with the NYUv2 training set, it improved a model's performance on real-world test data by +0.21 mIoU.
Superior Image Quality and VLN Success
MinecraftOcc significantly outperforms existing datasets in image clarity, achieving a Laplacian Variance (LV) score of 274,305 (compared to 57,369 for NYUv2). Furthermore, fine-tuning large vision-language models (like Qwen2.5-VL) on the MinecraftVLN dataset led to massive gains. For the "Next-Action" task, accuracy jumped from 0.19 (no training) to 0.72 (after SFT). The authors also successfully demonstrated a Gemini-2.5-Pro agent navigating a reconstructed scene in real-time to find specific objects based on text prompts.
Limitations and Future Work
Despite high-fidelity mods, Minecraft remains a blocky environment, leaving a residual visual "style" difference compared to photorealistic scans. Additionally, initial automated reconstructions can sometimes have "floating artifacts" or holes, requiring brief manual cleanup (though the authors note their SceneForge tool makes refining an automated reconstruction 7x faster than building from scratch). Future work aims to extend this pipeline from static indoor scenes to dynamic environments with moving objects or NPCs, and eventually scale to vast outdoor landscapes.
Summary updated