Reasoning in Vector Space: An Exploratory Study of Question Answering Moontae Lee

Xiaodong He, Wen-tau Yih, Jianfeng Gao, Li Deng, Paul Smolensky

TL;DR

This paper demonstrates that complex logical reasoning can be performed transparently in vector spaces using Tensor Product Representations (TPR). By disentangling semantic parsing from logical inference, the authors achieve near-perfect accuracy on the challenging bAbI reasoning benchmark, specifically solving "multi-relational" tasks like pathfinding that previously stumped end-to-end neural models.

The Problem: Black Box Models and Multi-Relational Gaps

While deep learning has made massive strides in Question Answering (QA), most modern models operate as "black boxes." When a model fails on a complex reasoning task, it is often unclear whether the failure happened because the model didn't understand the sentence's meaning (semantics) or because it couldn't follow the logical steps (reasoning).

The authors address two main gaps in current QA systems:

  1. Interpretability: There is a need for a framework where the "knowledge" stored in vectors is decodable and the reasoning steps are mathematically transparent.
  2. Multi-relational Reasoning: Most QA models handle simple transitive chains well (e.g., "A is in B, B is in C, so A is in C") but fail on tasks involving multiple independent relationships, such as spatial navigation or pathfinding.

Methodology: Tensor Product Representations (TPR)

The core of the paper is built on Tensor Product Representation (TPR), a mathematical framework for binding "fillers" (like an object or person) to "roles" (like a location or a relationship).

Key technical approaches include:

  • Binding and Unbinding: The model uses the outer product to bind a "filler" vector to a "role" vector. For example, to represent "Mary is in the kitchen," the vector for Mary is bound to the vector for kitchen. These can be "unbound" later using inner products or pseudo-inverses to retrieve specific information.
  • Containee-Container Models: For tasks involving movement and ownership, the authors use matrix multiplication to represent transitivity. If Mary moves to the garden, and she is carrying a ball, the ball’s location is updated by multiplying the ball-Mary tensor with the Mary-garden tensor.
  • Directional Matrices: For spatial tasks, the authors represent directions (North, South, East, West) as $d \times d$ non-singular matrices. Finding a path becomes a sequence of matrix transformations.
  • Disentangled Pipeline: To isolate reasoning, the authors use a standard dependency parser for semantic understanding. This allows them to focus purely on how the resulting vector representations perform during the logic phase.

Dataset: The bAbI Benchmark

The study evaluates this framework using the Facebook bAbI dataset, which consists of 20 different categories of synthetic reasoning tasks. These range from simple fact retrieval to complex deduction, induction, and pathfinding. While the vocabulary is small, the logical requirements are rigorous.

Examples of basic bAbI reasoning categories.

The dataset scales in difficulty, moving from single supporting facts to complex scenarios requiring the model to track indefinite knowledge, coreferences, and conjunctions.

Examples of advanced bAbI reasoning categories.

Key Insights and Results

The TPR-based models achieved near-perfect accuracy (100% or close) across almost all 20 bAbI categories.

  • The "Multi-Relational" Hurdle: The authors hypothesize that previous models (like early Memory Networks) failed on pathfinding and positional reasoning because those tasks are "multi-relational." Most other bAbI tasks are "uni-relational" (simple chains), which are easier for standard neural architectures to accidentally "solve" without true logical grounding.
  • Reasoning as Linear Algebra: The paper shows that common-sense inference rules (e.g., "An actor cannot be in two places at once") can be explicitly encoded as operations in vector space rather than just being "learned" implicitly.
  • Transparency is Possible: By using TPR, the authors show that we don't have to sacrifice interpretability for the power of distributed representations. You can look at the resulting tensors and mathematically "prove" why the model reached a specific answer.
  • Path Finding & Positional Reasoning: While previous state-of-the-art models struggled significantly with pathfinding (often scoring below 40%), the authors' approach handled it with high precision. It also successfully navigated spatial relationships that require understanding symmetry (e.g., if A is North of B, B is South of A).

Because the reasoning is based on mathematical properties of the matrices rather than just pattern matching, the model demonstrates a robust "common sense" that holds true across different story lengths.

Limitations and Future Directions

While the results are highly promising, the authors note several limitations:

  • Synthetic Language: The bAbI dataset uses very simple, unambiguous grammar. The "semantic parsing" side of their pipeline would need to be much more sophisticated to handle the noise and complexity of real-world natural language.
  • Manual vs. Learned: In this exploratory study, some of the directional matrices and relational structures were predefined. A major future direction is to develop "End-to-End TPR" models that can learn these structured matrices directly from data while maintaining their transparent properties.
  • Scaling: The study focuses on a small vocabulary. Scaling TPR to handle the vast number of entities and relations found in a massive knowledge graph like Freebase remains a significant challenge.

Summary updated

Abstract

Question answering tasks have shown remarkable progress with distributed vector representation. In this paper, we investigate the recently proposed Facebook bAbI tasks which consist of twenty different categories of questions that require complex reasoning. Because the previous work on bAbI are all end-to-end models, errors could come from either an imperfect understanding of semantics or in certain steps of the reasoning. For clearer analysis, we propose two vector space models inspired by Tensor Product Representation (TPR) to perform knowledge encoding and logical reasoning based on common-sense inference. They together achieve near-perfect accuracy on all categories including positional reasoning and path finding that have proved difficult for most of the previous approaches. We hypothesize that the difficulties in these categories are due to the multi-relations in contrast to uni-relational characteristic of other categories. Our exploration sheds light on designing more sophisticated dataset and moving one step toward integrating transparent and interpretable formalism of TPR into existing learning paradigms.
BibTeX
@inproceedings{he2016reasoning,
  author = {Xiaodong He and Wen-tau Yih and Jianfeng Gao and Li Deng and Paul Smolensky},
  title = {{Reasoning in Vector Space: An Exploratory Study of Question Answering Moontae Lee}},
  booktitle = {International Conference on Learning Representations},
  year = {2016},
  url = {http://arxiv.org/abs/1511.06426},
}
Figure