Learning Interpretable Spatial Operations in a Rich 3D Blocks World

Yonatan Bisk, Kevin J. Shih, Yejin Choi, Daniel Marcu

Introduction

Grounding natural language—connecting linguistic tokens to physical actions and spatial relationships—is a fundamental challenge in artificial intelligence. Most existing systems operate in simplified 2D grid worlds where movements are restricted to cardinal directions and fixed increments. However, human spatial reasoning is far more nuanced, involving continuous coordinates, rotations, and pragmatic concepts like "mirroring" or "balancing."

This research introduces a complex 3D blocks world dataset that pushes language grounding into a continuous 3D space. Alongside this dataset, the authors propose a modular neural architecture that discovers an interpretable library of spatial operations, such as "north" or "on top of," as discrete, reusable functions.

The 3D Blocks World Dataset

To bridge the gap between simple grids and real-world complexity, the authors developed a new 3D environment. This dataset significantly expands upon previous work (like the B16 2D dataset) by introducing:

  • 3D Structures: Blocks can be stacked into towers, arches, and other complex configurations.
  • Continuous Coordinates: Movement is no longer tied to a rigid grid, requiring real-valued positioning.
  • Rotations: Blocks can be angled and spun, introducing a new vocabulary of orientation (e.g., "clockwise," "45 degrees").
The 3D blocks world environment allows for continuous positioning and complex spatial arrangements.

The researchers collected approximately 13,000 new crowdsourced instructions across 100 3D world configurations, nearly doubling the available data for spatial grounding tasks.

Modular Neural Architecture

The proposed model is designed for interpretability, mapping natural language instructions and 3D world states to target coordinates $(x, y, z)$ and rotations $(\theta)$. The architecture consists of three key components:

  1. Language Encoding: Two separate Bi-LSTMs process instructions to identify the "argument" (the block to move) and the "operation" (the spatial action to perform).
  2. The Operation Bottleneck: Instead of a generic vector, the model selects from 32 "basis vectors" (latent operations) via a softmax layer. This forces the model to cluster similar linguistic concepts into discrete, interpretable operations.
  3. Spatial Prediction: A convolutional neural network (CNN) uses the selected operation and the current world state to predict the final placement.

Results and Emergent Interpretability

The model demonstrated high performance, achieving 91.3% accuracy in block identification on the 3D dataset, compared to 97.5% on the simpler 2D B16 dataset. In terms of precision, the average placement error was approximately 1.15 block lengths, with a remarkably low rotation error of only 3 degrees (0.058 radians).

Complex structures like these twin towers emerge from the model's ability to ground sophisticated spatial instructions.

A key finding was the "emergent interpretability" of the operation bottleneck. By visualizing the 32 basis vectors, researchers found the model automatically organized itself, assigning specific indices to concepts like "north," "southwest," and "on top of" without explicit supervision. For example, Operation 14 was found to consistently represent the "stacking" action. Furthermore, the model learned that operations are context-dependent; for instance, "to the right" adapts its magnitude to avoid pushing blocks off the table.

Conclusion and Future Work

This work demonstrates that richer environments lead to richer language use and more sophisticated grounding. While the model currently operates under relaxed physics constraints, it provides a strong foundation for future robotics applications. Future research will focus on handling implicit rotations and dynamic concept formation, where models can learn to define new landmarks or geometric shapes on the fly.

Summary updated

Abstract

In this paper, we study the problem of mapping natural language instructions to complex spatial actions in a 3D blocks world. We first introduce a new dataset that pairs complex 3D spatial operations to rich natural language descriptions that require complex spatial and pragmatic interpretations such as "mirroring", "twisting", and "balancing". This dataset, built on the simulation environment of Bisk, Yuret, and Marcu (2016), attains language that is significantly richer and more complex, while also doubling the size of the original dataset in the 2D environment with 100 new world configurations and 250,000 tokens. In addition, we propose a new neural architecture that achieves competitive results while automatically discovering an inventory of interpretable spatial operations (Figure 5)
BibTeX
@inproceedings{bisk2018learning,
  author = {Yonatan Bisk and Kevin J. Shih and Yejin Choi and Daniel Marcu},
  title = {{Learning Interpretable Spatial Operations in a Rich 3D Blocks World}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.12026},
}

Loading related content...

Figure