Introduction
Traditional attention mechanisms in neural networks, such as those used in Memory Networks or Neural Turing Machines, typically rely on discrete probability distributions over memory slots. While effective for simple retrieval, these mechanisms struggle with uncertainty and complex logical relations. When a query involves one-to-many relations (e.g., "Who are the children of Abraham Lincoln?") or multi-step reasoning, uncertainty compounds, making fixed-point representations inadequate.
This paper introduces the Gaussian Attention Model, which treats attention as a continuous probability distribution. By representing entities and relations as multivariate Gaussians, the model can explicitly account for semantic "spread" and uncertainty, leading to significant improvements in Knowledge Base (KB) embedding and complex question answering.
The Gaussian Attention Mechanism
The core innovation of this work is the replacement of the standard inner-product scoring function with a multivariate Gaussian likelihood. In traditional models, attention is a sharp point; in the Gaussian model, it is a distribution defined by a mean ($\mu$) and a covariance matrix ($\Sigma$).
The covariance matrix acts as a "degree of freedom," allowing the model to adjust its focus. A small variance indicates a specific, one-to-one mapping, while a large variance captures broader, one-to-many relationships. This flexibility allows the model to learn semantic subspaces where specific dimensions of the embedding space specialize in different attributes, such as nationality or position.
TransGaussian: Knowledge Base Embeddings
The authors apply this mechanism to KB embeddings through a model called TransGaussian. Unlike the classic TransE model, which uses simple vector translations ($subject + relation \approx object$), TransGaussian represents a relation $r$ as a mean offset $\delta_r$ and a diagonal covariance matrix $\Sigma_r$.
The score for a triplet $(s, r, o)$ is calculated as the log-likelihood of the object's vector under a Gaussian distribution centered at $v_s + \delta_r$. This approach naturally handles compositional relations: when traversing a path in a knowledge graph, the model performs Gaussian convolution, summing the means and variances along the path. This allows the "focus" of the attention to broaden mathematically as uncertainty increases along a chain of reasoning.
Question Answering Pipeline
The Gaussian framework provides an elegant mathematical basis for logical operations in natural language question answering. The pipeline consists of three main stages:
- Entity Recognition: Identifying key entities within the question.
- Relation Composition: An LSTM processes the question to predict weights for various atomic relations, forming a composite relation distribution.
- Conjunction: For questions with multiple constraints (e.g., "Who plays forward and plays for Barcelona?"), the model performs a product of Gaussian distributions. This Bayesian approach finds the intersection of constraints more robustly than heuristic scoring methods.
Experimental Evaluation
The model was evaluated on two primary datasets: WorldCup2014 (a custom dataset of football entities and relations) and WordNet.
The TransGaussian (COMP) model, trained with compositional paths, significantly outperformed the baseline TransE model across all metrics:
- Path Queries: TransGaussian achieved a Hits@1 (H@1) of 85.94%, compared to 77.39% for TransE.
- Conjunctive Queries: The performance gap was most pronounced here, with TransGaussian reaching 98.81% H@1 versus TransE’s 74.29%.
- Complex Reasoning: On specific queries like "Who are the defenders on the German national team?", TransGaussian achieved a perfect 100% H@1, while TransE struggled at 69.92%.
Conclusion
The Gaussian Attention Model offers a powerful alternative to traditional attention by incorporating a sense of uncertainty directly into the neural architecture. By leveraging the properties of Gaussian distributions—specifically convolution for path traversal and multiplication for logical conjunction—the model provides a mathematically rigorous and highly effective framework for reasoning over complex knowledge bases. Future work may explore the use of full covariance matrices to capture even more intricate correlations between semantic dimensions.
Summary updated