Modeling Semantics and Pragmatics of Spatial Prepositions via Hierarchical Common-Sense Primitives

Georgiy Platonov, Yifei Yang, Haoyu Wu, Jonathan Waxman, Marcus Hill, Lenhart SchubertACL 2021

TL;DR

This paper tackles the challenge of computationally modeling spatial prepositions (like "on," "in," "next to") by combining geometric scene analysis with hierarchical common-sense reasoning. They introduce a novel rule-based approach that decomposes complex spatial relations into simpler, interpretable "factors," outperforming pure neural network baselines and offering explainable judgments.

What Problem Does This Paper Address?

Spatial prepositions are notoriously tricky for AI. They're vague, have multiple meanings (polysemy), and their usage heavily depends on context and common-sense knowledge (pragmatics). While data-driven methods can learn statistical patterns, they often lack transparency, making it hard to understand why a model makes a certain judgment or to debug errors. This paper aims to build AI systems that can understand and use spatial prepositions more like humans do, by creating models that are not only accurate but also interpretable and grounded in a richer understanding of spatial relationships.

Methods

The authors propose a hierarchical, rule-based approach using "factors" to model spatial prepositions. Each preposition is represented as a tree where nodes compute increasingly complex spatial relations. These nodes combine simpler "factors" that encode basic geometric properties (like distance, size) and common-sense knowledge (like physical support, object orientation).

Key aspects of their method include:

  • Hierarchical Structure: Prepositions are broken down into a tree of simpler relations, allowing for a structured and interpretable model.
  • Factors: These are the building blocks, representing intuitive, often geometric, relationships between objects. Examples include "distance decay," "frame size rescale," and "supporting."
  • Soft Rules: Instead of hard logical rules, they use probabilistic computations (e.g., sigmoid functions) that allow for nuanced judgments and can be trained using gradient descent.
  • Explainability: The hierarchical structure naturally lends itself to generating explanations. By tracing the factors that contributed most to a final judgment, the system can explain why it believes a preposition holds true.
  • Rule-Based vs. Neural Baseline: They compare their primary rule-based models against a simpler neural network baseline (multi-layer perceptrons) to benchmark performance.

Datasets

The authors created their own dataset by generating 3D "room world" scenes using Blender. These scenes contain 10-30 objects each, with meronymy (part-of) relationships modeled. They collected human judgments on the spatial relationships between objects in these scenes. The paper mentions a total of under 7000 annotations for training and testing, with test sets comprising 20-30% of the data.

Key Insights and Ideas

  • Geometry + Common Sense is Key: Purely geometric models struggle with the nuances of spatial prepositions. Incorporating common-sense factors (like physical support for "on") significantly improves performance and interpretability.
  • Interpretability is Achievable: The hierarchical, factor-based approach provides a clear path to explaining AI's spatial judgments, moving beyond opaque "black box" models. This is crucial for building trust and enabling debugging in AI systems.
  • Context Matters: Different senses of prepositions (e.g., deictic vs. intrinsic "to the right of") need to be modeled, and the system selects the most appropriate sense based on context.
  • Pragmatics Influence Choice: Human annotators naturally preferred prepositions that disambiguate better or imply stronger relations, a pragmatic consideration the model aims to capture.

Key Results

  • The rule-based models generally performed as well as, and in some cases better than, the pure neural network baseline. For instance, they achieved higher accuracy on "in front of," "behind," and "touching."
  • For "to the right of," the rule-based model achieved 94% accuracy, while the NN baseline was also 94%. For "next to," the rule-based model achieved 95% accuracy compared to the NN baseline's 97%.
  • The system can generate explanations for its judgments. For example, it can explain "A is to the right of B" by stating "A is located on the right side of B," and further break down why that intrinsic relation holds (e.g., due to directional factors and proximity).

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the dataset size (under 7000 annotations) is relatively small, which limits the complexity of the neural baseline and potentially the robustness of the rule-based models.
  • Approximations: The system often relies on approximations like centroids and bounding boxes for object representation, which might miss subtle geometric details.
  • Generalization: While the "room world" is representative, generalizing to entirely novel environments or object types remains a challenge.
  • Future Work: The authors plan to integrate their models into dialogue systems to evaluate explainability in a conversational setting. They are also working on expanding the dataset with more scenes, including outdoor environments, and exploring how scale differences affect prepositional judgments.

Summary updated

BibTeX
@inproceedings{platonov2021modeling,
  author = {Georgiy Platonov and Yifei Yang and Haoyu Wu and Jonathan Waxman and Marcus Hill and Lenhart Schubert},
  title = {{Modeling Semantics and Pragmatics of Spatial Prepositions via Hierarchical Common-Sense Primitives}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure