What a neural language model tells us about spatial relations

Mehdi Ghanimifard, Simon DobnikNAACL 2019

TL;DR

This paper explores how neural language models capture the nuances of spatial relations, distinguishing between their geometric and functional aspects. By analyzing the model's prediction patterns (perplexity), they develop novel vector representations that effectively capture these semantic distinctions, outperforming existing methods on analogy and odd-one-out tasks.

What Problem Does This Paper Address?

Understanding and generating descriptions of spatial relationships between objects (e.g., "the chair is to the left of the table") is crucial for tasks like robotics and image captioning. These descriptions rely on two types of knowledge: geometric (where objects are) and functional (how objects interact). While neural language models are adept at processing text, it's unclear how well they encode these distinct types of spatial knowledge and how to extract it. This paper aims to bridge this gap by investigating what semantic knowledge about spatial relations is captured by generative neural language models and how to represent and evaluate it.

Methods

The core of the paper's methodology lies in using perplexity as a measure of how well a trained neural language model predicts a given sequence of words. The authors hypothesize that functionally biased spatial relations will be more predictable by a text-based model than geometrically biased ones.

  1. Perplexity-Based Vector Representations (P-vectors): They create "context templates" by taking sentences describing spatial relations and systematically replacing the original spatial relation with others. The perplexity of the language model on these modified sentences reveals how "swappable" a relation is within a given context. This process generates a matrix where each row represents a spatial relation and each column represents a context. These perplexity values are then normalized to create "perplexity vectors" (P-vectors) for each spatial relation.
  2. LM-Vectors: They also extract word embeddings directly from the trained recurrent neural language model (LSTM). For multi-word relations, they sum the embeddings of individual words.
  3. GloVe Embeddings: They use pre-trained GloVe embeddings (both from the Visual Genome corpus and Common Crawl) for comparison.

Datasets

  • Visual Genome Region Description Corpus: Used for training the neural language model and for extracting spatial relation contexts. This dataset is chosen specifically for its "spatial usages of prepositions."
  • Human Judgments:
    • Word Association Judgments (De Deyne et al., 2018): Used to compare model representations with general word associations.
  • New Word Similarity Judgments (Amazon Mechanical Turk): A custom dataset where participants rated the similarity of spatial relation pairs.
  • Spatial Templates (Logan and Sadler, 1996): A dataset representing geometric locations of objects.

Key Insights and Ideas

  • Functional vs. Geometric Bias: The paper strongly suggests that functional and geometric knowledge of spatial relations are in complementary distribution. This means that when one is strong, the other might be weaker, and vice-versa.
  • Perplexity as a Semantic Probe: Perplexity, a measure of a language model's prediction difficulty, can be effectively used to uncover semantic distinctions between spatial relations, particularly their functional biases.
  • P-vectors Capture Fine-Grained Semantics: The generated P-vectors demonstrate an ability to capture subtle semantic differences between spatial relations, even those that are geometrically similar but functionally distinct (e.g., "above" vs. "over").
  • Context Matters: The choice of description for a spatial relation is context-dependent, highlighting the dynamic nature of meaning.
  • Implications for Image Captioning: The findings suggest that even with access to visual features, language models' understanding of spatial relations is heavily influenced by textual context, which has implications for how we evaluate and build image captioning systems.

Key Results

  • Clustering: P-vectors, when clustered, group semantically similar spatial relations (e.g., synonyms like "above, over") and antonyms (e.g., "on the top of, on the bottom of").
  • Analogical Reasoning: P-vectors performed competitively, and slightly better than LM-embeddings, on multi-word analogy tasks, suggesting they capture functional associations well. LM-embeddings performed best on single-word analogies.
  • Odd-One-Out: P-vectors achieved good accuracy in identifying the odd relation out of three, demonstrating their ability to distinguish between functional and geometric distinctions.
  • Human Similarity Correlation: P-vectors and LM-vectors showed a better correlation with word association judgments (WA) than with direct word similarity judgments (WS), suggesting they capture functional aspects more effectively.

Limitations and Future Work

  • Limitations:
    • The study acknowledges that current language models are primarily trained on textual data, and while they can infer geometric properties, they don't inherently "see" the scene. This means image captioning systems still need to ground descriptions in visual features.
  • The correlation with human WS judgments was weaker than with WA, possibly due to participants in WS tasks focusing more on geometric dimensions.
  • Future Work:
    • Investigate how to use the knowledge of spatial relation bias to improve captioning tasks, especially with spatial word substitutions.
  • Explore how functional knowledge complements visual knowledge in language generation.
  • Utilize more advanced contextual embeddings like ELMo and BERT.
  • Compare language models for spatial descriptions across different pragmatic tasks (e.g., navigation instructions vs. general image captioning), expecting stronger geometric bias in tasks requiring precise localization.

Summary updated

BibTeX
@inproceedings{ghanimifard2019what,
  author = {Mehdi Ghanimifard and Simon Dobnik},
  title = {{What a neural language model tells us about spatial relations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure