Abstract
In this paper, we study the problem of sequentially building houses in the game of Minecraft, and demonstrate that learning the ordering can make for more effective autoregressive models. Given a partially built house made by a human player, our system tries to place additional blocks in a human-like manner to complete the house. We introduce a new dataset, HouseCraft, for this new task. HouseCraft contains the sequential order in which 2,500 Minecraft houses were built from scratch by humans. The human action sequences enable us to learn an order-aware generative model called Voxel-CNN. In contrast to many generative models where the sequential generation ordering either does not matter (e.g. holistic generation with GANs), or is manually/arbitrarily set by simple rules (e.g. raster-scan order), our focus is on an ordered generation that imitates humans. To evaluate if a generative model can accurately predict human-like actions, we propose several novel quantitative metrics. We demonstrate that our Voxel-CNN model is simple and effective at this creative task, and can serve as a strong baseline for future research in this direction. The HouseCraft dataset and code with baseline models will be made publicly available.
Introduction
Researchers from Facebook AI Research introduce 3D-Craft (also referred to as HouseCraft), a large-scale dataset of 2,500 Minecraft houses that records the exact sequence of blocks placed by human players. They leverage this data to train VoxelCNN, an autoregressive model that learns to generate 3D structures by mimicking human building patterns, outperforming standard heuristic-based approaches like raster-scanning.
The Importance of Construction Order
In 2D image generation, we often follow a simple "raster scan" (top-to-bottom, left-to-right) order. However, generating 3D objects in the real world is fundamentally constrained by the construction process. You cannot build a roof before the walls, and you must account for physical constraints like gravity and accessibility.
Most existing 3D generative models focus solely on the final artifact (the completed voxel grid or point cloud) and ignore the order of operations. While one could try to simulate complex physics to find a valid building order, this paper argues that imitating human behavior is a more efficient proxy. Humans implicitly follow physical constraints and semantic logic (e.g., finishing one wall before starting the next) when they build.
VoxelCNN: An Order-Aware Generative Model
The authors propose VoxelCNN, an order-aware generative model designed to predict the next building action based on the current state of the object.
- Autoregressive Modeling: The model predicts the joint distribution of actions as a product of conditional distributions. Essentially, it asks: "Given what has been built so far, where is the next block, and what material is it?"
- Two-Stream Architecture: To capture both fine details and the "big picture," the model uses two branches: a high-resolution Local Encoding (a 3D patch centered on the last placed block) and a lower-resolution Global Encoding (a larger-scale binary occupancy map).
- Factorized Prediction: The model first predicts the location of the next block within a local neighborhood and then, conditioned on that location, predicts the material type.
- Temporal Stacking: To give the model "memory," it stacks the states of the previous few steps, allowing it to recognize ongoing patterns like building a straight line or a repetitive fence.
The 3D-Craft Dataset
The primary contribution is the 3D-Craft dataset, created using a custom Minecraft-compatible server (Cuberite). It consists of 2,500 unique houses built by over 200 human players, totaling 2.8 million individual building steps across 256 different material types. Unlike ShapeNet, which provides static CAD models, 3D-Craft provides the full temporal sequence of construction.
Key Insights and Results
- The "Natural Order" Advantage: Human building order is semantically meaningful. Humans tend to finish sub-structures (like a wall) before moving on. Heuristic orders like raster-scanning break this continuity, making the generation task harder for neural networks.
- Spatial Locality: Data analysis showed that ~70% of blocks are placed within just 1 unit of the previous block (see Figure 6a). This justifies the model's focus on local neighborhoods for prediction.
- Quantitative Success: VoxelCNN achieved 62.7% ACC@1 (predicting the exact next block location and type), significantly higher than the Nearest Neighbor baseline (42.9%) and LSTM (32.1%). In terms of Perplexity, VoxelCNN scored 3.24, outperforming the Learned Raster-Scan (4.44).
- Qualitative Success: The model demonstrated "intuitive" behaviors, such as knowing to switch materials when finishing a wall to start a roof, and avoiding placing floating blocks in mid-air.
Limitations and Future Work
- Local Window Constraint: The current model assumes the next block is within a small local window. While this covers 90% of human actions, it fails to capture "large jumps" (e.g., moving from the ground floor to start the chimney).
- Voxel-by-Voxel Speed: Building a complex house one block at a time is computationally expensive. Future work could move toward part-based generation (placing entire walls or windows at once).
- Semantic Labeling: The current dataset lacks high-level semantic labels. Adding these labels could enable more controllable generation (e.g., "add a balcony here").
Summary updated