Quantum Statistics-Inspired Neural Attention

Aristotelis Charalampous, Sotirios Chatzis

Abstract

Standard neural attention mechanisms often treat source tokens as independent entities, potentially overlooking higher-order dependencies and structural ambiguities inherent in natural language. This paper introduces a novel attention mechanism inspired by quantum statistics that replaces categorical attention weights with an "Attention Density Matrix" (ADM). By explicitly modeling the uncertainty between pairs of source elements, the proposed Multiplicative Quantum-inspired Transformer (MQT) captures latent dynamics without significant computational overhead. Empirical results on machine translation tasks demonstrate improved BLEU scores and superior handling of rare words compared to standard soft-attention and traditional structured attention models.

Introduction

In sequence-to-sequence models, Soft-Attention (SA) serves as a point-wise selection mechanism, assigning probabilities to source tokens under the assumption of independence. However, language is rich with higher-order dependencies, such as phrases and complex syntax, which are often missed by isolated token analysis. While "structured attention" methods using Conditional Random Fields (CRFs) attempt to address this, they are often computationally prohibitive and rely on restrictive Markovian assumptions that limit their expressive power.

Methodology: The Attention Density Matrix

To efficiently capture higher-order relationships, this work draws on the concept of a Density Matrix from quantum statistics. In a quantum system, a "mixed state" represents an inherent inability to discern between individual states, a property the authors adapt to model linguistic ambiguity.

The authors propose the Attention Density Matrix (ADM):

  • Diagonal Elements: Correspond to standard Soft-Attention scores, representing the probability of attending to individual tokens.
  • Off-Diagonal Elements: Represent the model's uncertainty or the latent dynamics shared between a pair of source tokens.

The model computes a 3D tensor capturing correlations between all source encoding pairs. Two variants are explored: Multiplicative (MQT) and Additive (AQT). The context vector is derived by applying a softmax over the row-wise mean of the ADM. Notably, the implementation avoids strict positive-semidefinite constraints, allowing for a single extra forward-propagation step, maintaining efficiency comparable to standard Soft-Attention.

Experimental Setup

The models were evaluated on two Machine Translation benchmarks:

  • IWSLT’15 English ↔ Vietnamese (En↔Vi): Approximately 133K training pairs from transcribed TED talks.
  • WMT’16 English ↔ Romanian (En↔Ro): Approximately 400K parallel sentences.

These relatively small datasets were chosen to test the generalization capabilities of the proposed mechanism in data-constrained environments.

Key Results

The Multiplicative (MQT) variant consistently outperformed both the baseline Soft-Attention and traditional Structured Attention models across all translation directions.

BLEU scores for different attention mechanisms across IWSLT'15 (En-Vi) and WMT'16 (En-Ro) benchmarks.

As shown in the results, MQT achieved a BLEU score of 25.34 on En→Vi (test), compared to 24.11 for the baseline. On the more challenging En→Ro task, MQT reached 16.95 BLEU, significantly higher than the 15.31 achieved by the baseline and the 7.03 by Structured Attention.

Beyond aggregate scores, MQT showed significant improvements in specific linguistic areas:

  • Rare Word Handling: MQT exhibited only a 0.28% deviation from ground-truth rare word frequency on En→Vi, whereas the baseline and structured attention deviated by 4.82% and 11.57%, respectively.
  • Qualitative Coherence: Manual inspection revealed that MQT translations were less prone to "missing token" errors, such as dropped verbs or adjectives, which frequently plague standard models when dealing with complex phrases.

Discussion and Future Work

While the ADM effectively captures second-order interactions, it does not currently handle higher-rank hierarchical dependencies (e.g., triplets of tokens). Furthermore, the use of row-wise mean-pooling is a computational approximation; a more rigorous spectral decomposition approach currently faces numerical instability during back-propagation. Future research will focus on stabilizing these decompositions to enable true high-rank matrix approximations and exploring the mechanism's utility in other sequence-to-sequence tasks.

Summary updated

Abstract

Sequence-to-sequence (encoder-decoder) models with attention constitute a cornerstone of deep learning research, as they have enabled unprecedented sequential data modeling capabilities. This effectiveness largely stems from the capacity of these models to infer salient temporal dynamics over long horizons; these are encoded into the obtained neural attention (NA) distributions. However, existing NA formulations essentially constitute point-wise selection mechanisms over the observed source sequences; that is, attention weights computation relies on the assumption that each source sequence element is independent of the rest. Unfortunately, although convenient, this assumption fails to account for higher-order dependencies which might be prevalent in real-world data. This paper addresses these limitations by leveraging Quantum-Statistical modeling arguments. Specifically, our work broadens the notion of NA, by attempting to account for the case that the NA model becomes inherently incapable of discerning between individual source elements; this is assumed to be the case due to higher-order temporal dynamics. On the contrary, we postulate that in some cases selection may be feasible only at the level of pairs of source sequence elements. To this end, we cast NA into inference of an attention density matrix (ADM) approximation. We derive effective training and inference algorithms, and evaluate our approach in the context of a machine translation (MT) application. We perform experiments with challenging benchmark datasets. As we show, our approach yields favorable outcomes in terms of several evaluation metrics.
BibTeX
@misc{charalampous2018quantum,
  author = {Aristotelis Charalampous and Sotirios Chatzis},
  title = {{Quantum Statistics-Inspired Neural Attention}},
  year = {2018},
  eprint = {1809.06205},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1809.06205},
}
Figure