Learning Interpretable Spatial Operations in a Rich 3D Blocks World

Yonatan Bisk, Kevin J. Shih, Yejin Choi, Daniel Marcu

Abstract

In this paper, we study the problem of mapping natural language instructions to complex spatial actions in a 3D blocks world. We introduce a new dataset that pairs complex 3D spatial operations with rich natural language descriptions requiring interpretations such as "mirroring", "twisting", and "balancing". This dataset doubles the size of previous 2D environments and features significantly more complex language. We also propose a neural architecture that achieves competitive results while automatically discovering an inventory of interpretable spatial operations.

Bridging the Gap Between Simulation and Reality

Building AI agents capable of following natural language instructions in physical environments is a long-standing goal in AI. However, many current approaches rely on simplified 2D grid worlds where spatial terms like "right" are reduced to fixed offsets. In the real world, spatial relationships are context-dependent, influenced by object size, perspective, and environment boundaries.

This research addresses this gap by introducing a 3D blocks world characterized by continuous coordinates and rotations. This environment forces models to learn richer spatial reasoning beyond simple grid movements.

An example of a block movement task in the 3D environment, showing the initial state and the result of a spatial operation.

A Richer 3D Dataset

The authors introduce a new 3D dataset that significantly expands upon the original B16 2D blocks world. It includes 100 new world configurations and approximately 13,000 crowdsourced instructions. Unlike its predecessor, this dataset features complex 3D structures like towers, arches, and intricate human-designed patterns. These configurations necessitate an understanding of advanced concepts such as "balancing," "mirroring," and "twisting," leading to much more pragmatic and complex linguistic references from human annotators.

A complex 3D structure from the new dataset, illustrating the increased difficulty and richness of the environment.

Modular Architecture for Interpretability

The proposed model is designed for both performance and interpretability, consisting of three main stages:

  1. Language Encoding: Dual bi-LSTMs process instructions to identify the target block (the "argument") and the intended spatial operation.
  2. Operation Selection: The model selects from an inventory of 32 learned "basis vectors" (spatial functions) via a latent softmax. This architectural bottleneck forces the model to cluster similar linguistic concepts into shared, interpretable operators.
  3. Spatial Prediction: The selected operator is applied to the world state, and a Convolutional Neural Network (CNN) predicts the final 3D coordinates and rotation for the block.

Key Insights and Results

  • Interpretable Bottlenecks: By restricting the model to a discrete set of 32 operations, the researchers created a window into the model's logic, allowing them to see which phrases trigger specific mathematical operations.
  • Spatial Accuracy: The model achieved 91.3% accuracy in block identification. While the average placement error (1.15 block lengths) is higher than in 2D tasks, it reflects the increased complexity of the 3D domain.
  • Mastering Rotations: The model successfully learned to predict rotations with a mean error of only 0.058 radians (approx. 3 degrees).
  • Complexity and Language: The move to 3D fundamentally changed how humans describe tasks, introducing pragmatic references like "mirroring the other side" that challenge standard grounding models.

Conclusion

Current limitations include the use of relaxed physics (e.g., temporary floating blocks) and difficulty with implied rotations. Future research will focus on handling compound instructions by composing multiple learned operators together.

Summary updated

Abstract

In this paper, we study the problem of mapping natural language instructions to complex spatial actions in a 3D blocks world. We first introduce a new dataset that pairs complex 3D spatial operations to rich natural language descriptions that require complex spatial and pragmatic interpretations such as "mirroring", "twisting", and "balancing". This dataset, built on the simulation environment of Bisk, Yuret, and Marcu (2016), attains language that is significantly richer and more complex, while also doubling the size of the original dataset in the 2D environment with 100 new world configurations and 250,000 tokens. In addition, we propose a new neural architecture that achieves competitive results while automatically discovering an inventory of interpretable spatial operations (Figure 5)
BibTeX
@inproceedings{bisk2018learning,
  author = {Yonatan Bisk and Kevin J. Shih and Yejin Choi and Daniel Marcu},
  title = {{Learning Interpretable Spatial Operations in a Rich 3D Blocks World}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.12026},
}
Figure