Abstract
In this paper, we study the problem of mapping natural language instructions to complex spatial actions in a 3D blocks world. We introduce a new dataset that pairs complex 3D spatial operations with rich natural language descriptions requiring interpretations such as "mirroring", "twisting", and "balancing". This dataset doubles the size of previous 2D environments and features significantly more complex language. We also propose a neural architecture that achieves competitive results while automatically discovering an inventory of interpretable spatial operations.
Bridging the Gap Between Simulation and Reality
Building AI agents capable of following natural language instructions in physical environments is a long-standing goal in AI. However, many current approaches rely on simplified 2D grid worlds where spatial terms like "right" are reduced to fixed offsets. In the real world, spatial relationships are context-dependent, influenced by object size, perspective, and environment boundaries.
This research addresses this gap by introducing a 3D blocks world characterized by continuous coordinates and rotations. This environment forces models to learn richer spatial reasoning beyond simple grid movements.
A Richer 3D Dataset
The authors introduce a new 3D dataset that significantly expands upon the original B16 2D blocks world. It includes 100 new world configurations and approximately 13,000 crowdsourced instructions. Unlike its predecessor, this dataset features complex 3D structures like towers, arches, and intricate human-designed patterns. These configurations necessitate an understanding of advanced concepts such as "balancing," "mirroring," and "twisting," leading to much more pragmatic and complex linguistic references from human annotators.
Modular Architecture for Interpretability
The proposed model is designed for both performance and interpretability, consisting of three main stages:
- Language Encoding: Dual bi-LSTMs process instructions to identify the target block (the "argument") and the intended spatial operation.
- Operation Selection: The model selects from an inventory of 32 learned "basis vectors" (spatial functions) via a latent softmax. This architectural bottleneck forces the model to cluster similar linguistic concepts into shared, interpretable operators.
- Spatial Prediction: The selected operator is applied to the world state, and a Convolutional Neural Network (CNN) predicts the final 3D coordinates and rotation for the block.
Key Insights and Results
- Interpretable Bottlenecks: By restricting the model to a discrete set of 32 operations, the researchers created a window into the model's logic, allowing them to see which phrases trigger specific mathematical operations.
- Spatial Accuracy: The model achieved 91.3% accuracy in block identification. While the average placement error (1.15 block lengths) is higher than in 2D tasks, it reflects the increased complexity of the 3D domain.
- Mastering Rotations: The model successfully learned to predict rotations with a mean error of only 0.058 radians (approx. 3 degrees).
- Complexity and Language: The move to 3D fundamentally changed how humans describe tasks, introducing pragmatic references like "mirroring the other side" that challenge standard grounding models.
Conclusion
Current limitations include the use of relaxed physics (e.g., temporary floating blocks) and difficulty with implied rotations. Future research will focus on handling compound instructions by composing multiple learned operators together.
Summary updated