Probabilistic Reasoning via Deep Learning: Neural Association Models

Quan Liu, Hui Jiang, Zhen-Hua Ling, Si Wei, Yu HuarXiv 2016 · OpenReview

TL;DR

This paper introduces Neural Association Models (NAMs), a deep learning framework for probabilistic reasoning that models the conditional probability between any two events. NAMs, implemented as Deep Neural Networks (DNNs) and Relation-Modulated Neural Networks (RMNNs), outperform traditional methods on tasks like textual entailment and knowledge base completion, with RMNNs showing promise in knowledge transfer.

What Problem Does This Paper Address?

Traditional probabilistic reasoning methods, like Bayesian Networks, struggle with the massive scale and complexity of real-world domains, often becoming computationally intractable. While recent advances in distributed representations and knowledge graph embeddings have shown promise, human-like reasoning, especially commonsense reasoning, remains a significant challenge. This paper aims to bridge this gap by proposing a deep learning approach that can model the probabilistic associations between events in a more scalable and flexible manner, without requiring explicit dependency structures.

Methods

The core of the proposed approach is the Neural Association Model (NAM). NAMs leverage deep neural networks to model the conditional probability $Pr(E_2|E_1)$ between two events, $E_1$ (antecedent) and $E_2$ (consequent). Events are first embedded into a low-dimensional continuous space. The NAM then takes the embedding of $E_1$ as input and outputs the conditional probability of $E_2$.

Two specific NAM architectures are explored:

  • Deep Neural Networks (DNNs): A standard feedforward neural network architecture is used. For multi-relational data, like knowledge graph triples $(e_i, r_k, e_j)$, the input is formed by combining the embeddings of the head entity and relation, and the network predicts the probability of the tail entity.
  • Relation-Modulated Neural Networks (RMNNs): This architecture is designed to be more effective for multi-relational data. It incorporates a "relation code" into each hidden layer of the network, allowing the model to learn relation-specific transformations. This design is hypothesized to be beneficial for knowledge transfer.

The models are trained using maximum likelihood estimation via stochastic gradient descent, with negative sampling used to generate training data.

Datasets

The paper evaluates NAMs on several established datasets and introduces a new dataset for commonsense reasoning:

  • SNLI dataset: Used for recognizing textual entailment.
  • WN11 and FB13 datasets: Used for triple classification in multi-relational knowledge bases (WordNet and FreeBase).
  • ConceptNet KB: Used to construct a new commonsense reasoning dataset, CN14. This dataset is designed to answer commonsense questions by calculating association probabilities between entities and relations.
  • Winograd Schema Challenge (WSC) problems: A subset of WSC problems were manually labeled to create a cause-effect dataset for evaluating commonsense reasoning.

Key Insights and Ideas

  • Deep Learning for Probabilistic Reasoning: The paper demonstrates that deep neural networks can effectively model probabilistic associations between events, offering a powerful alternative to traditional graphical models.
  • Flexibility of NAMs: NAMs can model various types of associations (cause-effect, entailment, etc.) by simply adjusting how events are represented and how the models are trained. The meaning of the conditional probabilities is application-dependent.
  • RMNNs for Knowledge Transfer: The relation-modulated architecture (RMNN) shows a significant advantage in knowledge transfer. It can adapt to new relations with only a few training samples without degrading performance on previously learned relations, a capability that standard DNNs struggle with.
  • Scalability Potential: By avoiding explicit dependency structures and leveraging distributed representations, NAMs offer a more scalable approach to probabilistic reasoning in large domains.

Key Results

  • Textual Entailment: The proposed DNN-based NAM achieved an accuracy of 84.7% on the SNLI dataset, significantly outperforming baseline methods (up to 75.0%).
  • Triple Classification: Both DNNs and RMNNs achieved comparable and superior performance on WN11 and FB13 datasets compared to existing methods, including Neural Tensor Networks (NTN). RMNN showed a 3.7% and 1.9% absolute improvement on WN11 and FB13 respectively.
  • Commonsense Reasoning (CN14): NAMs (DNN and RMNN) outperformed NTN on the CN14 dataset, achieving accuracies of 85.7% and 87.1% respectively, compared to NTN's 84.6%.
  • Knowledge Transfer: RMNN demonstrated superior performance in knowledge transfer, significantly improving accuracy on a new relation with limited samples while maintaining performance on existing relations. In contrast, DNN's performance on original relations degraded.
  • Winograd Schema Challenge: NAMs achieved approximately 60% accuracy on a cause-effect dataset derived from WSC problems, indicating their potential for commonsense reasoning.

Limitations and Future Work

  • Data Hungry: The paper acknowledges that deep learning models, including NAMs, are data-hungry, making data collection a challenging aspect for real-world applications.
  • Winograd Schema Performance: While NAMs show potential for WSC problems, the reported 60% accuracy suggests there is still significant room for improvement to achieve human-level performance.
  • Future Work: The authors suggest further exploration of RMNNs for knowledge transfer and applying NAMs to more complex commonsense reasoning tasks. They also highlight the need for more efficient methods for collecting associative data.

Summary updated

Abstract

In this paper, we propose a new deep learning approach, called neural association model (NAM), for probabilistic reasoning in artificial intelligence. We propose to use neural networks to model association between any two events in a domain. Neural networks take one event as input and compute a conditional probability of the other event to model how likely these two events are to be associated. The actual meaning of the conditional probabilities varies between applications and depends on how the models are trained. In this work, as two case studies, we have investigated two NAM structures, namely deep neural networks (DNN) and relation-modulated neural nets (RMNN), on several probabilistic reasoning tasks in AI, including recognizing textual entailment, triple classification in multi-relational knowledge bases and commonsense reasoning. Experimental results on several popular datasets derived from WordNet, FreeBase and ConceptNet have all demonstrated that both DNNs and RMNNs perform equally well and they can significantly outperform the conventional methods available for these reasoning tasks. Moreover, compared with DNNs, RMNNs are superior in knowledge transfer, where a pre-trained model can be quickly extended to an unseen relation after observing only a few training samples. To further prove the effectiveness of the proposed models, in this work, we have applied NAMs to solving challenging Winograd Schema (WS) problems. Experiments conducted on a set of WS problems prove that the proposed models have the potential for commonsense reasoning.
BibTeX
@misc{liu2016probabilistic,
  author = {Quan Liu and Hui Jiang and Zhen-Hua Ling and Si Wei and Yu Hu},
  title = {{Probabilistic Reasoning via Deep Learning: Neural Association Models}},
  year = {2016},
  eprint = {1603.07704},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1603.07704},
}
Figure