Abstract
Standard neural attention mechanisms often treat source tokens as independent entities, potentially overlooking higher-order dependencies and structural ambiguities inherent in natural language. This paper introduces a novel attention mechanism inspired by quantum statistics that replaces categorical attention weights with an "Attention Density Matrix" (ADM). By explicitly modeling the uncertainty between pairs of source elements, the proposed Multiplicative Quantum-inspired Transformer (MQT) captures latent dynamics without significant computational overhead. Empirical results on machine translation tasks demonstrate improved BLEU scores and superior handling of rare words compared to standard soft-attention and traditional structured attention models.
Introduction
In sequence-to-sequence models, Soft-Attention (SA) serves as a point-wise selection mechanism, assigning probabilities to source tokens under the assumption of independence. However, language is rich with higher-order dependencies, such as phrases and complex syntax, which are often missed by isolated token analysis. While "structured attention" methods using Conditional Random Fields (CRFs) attempt to address this, they are often computationally prohibitive and rely on restrictive Markovian assumptions that limit their expressive power.
Methodology: The Attention Density Matrix
To efficiently capture higher-order relationships, this work draws on the concept of a Density Matrix from quantum statistics. In a quantum system, a "mixed state" represents an inherent inability to discern between individual states, a property the authors adapt to model linguistic ambiguity.
The authors propose the Attention Density Matrix (ADM):
- Diagonal Elements: Correspond to standard Soft-Attention scores, representing the probability of attending to individual tokens.
- Off-Diagonal Elements: Represent the model's uncertainty or the latent dynamics shared between a pair of source tokens.
The model computes a 3D tensor capturing correlations between all source encoding pairs. Two variants are explored: Multiplicative (MQT) and Additive (AQT). The context vector is derived by applying a softmax over the row-wise mean of the ADM. Notably, the implementation avoids strict positive-semidefinite constraints, allowing for a single extra forward-propagation step, maintaining efficiency comparable to standard Soft-Attention.
Experimental Setup
The models were evaluated on two Machine Translation benchmarks:
- IWSLT’15 English ↔ Vietnamese (En↔Vi): Approximately 133K training pairs from transcribed TED talks.
- WMT’16 English ↔ Romanian (En↔Ro): Approximately 400K parallel sentences.
These relatively small datasets were chosen to test the generalization capabilities of the proposed mechanism in data-constrained environments.
Key Results
The Multiplicative (MQT) variant consistently outperformed both the baseline Soft-Attention and traditional Structured Attention models across all translation directions.
As shown in the results, MQT achieved a BLEU score of 25.34 on En→Vi (test), compared to 24.11 for the baseline. On the more challenging En→Ro task, MQT reached 16.95 BLEU, significantly higher than the 15.31 achieved by the baseline and the 7.03 by Structured Attention.
Beyond aggregate scores, MQT showed significant improvements in specific linguistic areas:
- Rare Word Handling: MQT exhibited only a 0.28% deviation from ground-truth rare word frequency on En→Vi, whereas the baseline and structured attention deviated by 4.82% and 11.57%, respectively.
- Qualitative Coherence: Manual inspection revealed that MQT translations were less prone to "missing token" errors, such as dropped verbs or adjectives, which frequently plague standard models when dealing with complex phrases.
Discussion and Future Work
While the ADM effectively captures second-order interactions, it does not currently handle higher-rank hierarchical dependencies (e.g., triplets of tokens). Furthermore, the use of row-wise mean-pooling is a computational approximation; a more rigorous spectral decomposition approach currently faces numerical instability during back-propagation. Future research will focus on stabilizing these decompositions to enable true high-rank matrix approximations and exploring the mechanism's utility in other sequence-to-sequence tasks.
Summary updated