Attention improves concentration when learning node embeddings

Matthew Dippel, Adam Kiezun, Tanay Mehta, Ravi Sundaram, Srikanth Thirumalai, Akshar VarmaarXiv 2020 · arXiv

Abstract

Attention mechanisms have become a cornerstone of modern deep learning, yet their specific advantages in learning node embeddings for e-commerce graphs remain under-explored. This paper introduces AttEST (Attention Embeddings for Short Texts), a generative model that treats search queries and products as vectors in a latent space. We provide a theoretical justification for attention's success, demonstrating that it functions as a variance reduction tool akin to a Best Linear Unbiased Estimator (BLUE). Empirical results on large-scale Amazon datasets show that attention-based models significantly outperform complex architectures like LSTMs by effectively filtering linguistic noise and concentrating on core product intent.

Introduction

In the domain of e-commerce, "query reformulation"—identifying different search strings that lead to the same product purchase—is a critical task. This can be modeled as a link prediction problem in a query-product graph where nodes represent queries and edges represent shared purchase behavior. However, the sheer scale of commercial vocabularies and the inherent noise in short search queries pose significant challenges. While large language models like BERT are powerful, their computational cost is often prohibitive at scale. This research investigates why simpler attention-based models often yield superior results, proposing a "concentration theory" where attention serves to de-noise input signals.

Methodology: The AttEST Model

The authors propose AttEST (Attention Embeddings for Short Texts), a framework designed to handle the high-cardinality vocabulary of e-commerce search queries. Instead of full word embeddings, the model utilizes a feedforward network to learn embeddings for trigrams (sequences of three characters or words). A final query vector is produced by an attention mechanism that assigns weights to these trigrams, allowing the model to focus on the most salient parts of the input.

Theoretical Framework: Attention as BLUE

The model's loss function is derived from maximizing the Point-wise Mutual Information (PMI) between related queries in the graph. The authors provide a formal proof that the resulting PMI matrix is low-rank, which mirrors the structural properties found in successful word embedding models like word2vec. This provides a solid theoretical foundation for using simple dot-product similarity to measure query relevance.

The core of their theoretical contribution is the application of the Gauss-Markov theorem to attention. They demonstrate that attention weights correlate with the Best Linear Unbiased Estimator (BLUE). In this framework, as a query grows in length, it accumulates "linguistic noise"—extra words that don't change the underlying product intent. Attention allows the model to "concentrate" on the most informative terms (the signal) while discounting discursive or redundant terms (the noise). This variance reduction is what allows simple models to outperform more complex, sequence-dependent architectures.

Experimental Results

The model was evaluated on two datasets from Amazon US: a large-scale Electronics category and a denser Top-25 Clusters dataset.

Performance comparison across different model architectures showing the superiority of attention-based models.

As shown in the table above, the Attention-based model consistently outperforms traditional LSTM architectures. In the Top-25 Clusters dataset, the Attention model achieved an F1 score of 65.3%, compared to 44.4% for a standard LSTM—a relative improvement of over 20%.

Empirical Validation of Concentration Theory

To validate the "concentration" hypothesis, the authors compared learned attention weights against theoretical BLUE weights.

Empirical validation showing (a) the alignment between attention weights and BLUE weights, and (b) the relationship between experimental and theoretical variance.

The results in Figure 1a demonstrate a strong correlation: attention weights typically trend downward as a query progresses, reflecting the intuition that primary product descriptors (e.g., "iPhone") usually appear earlier than secondary details (e.g., "white 32gb"). The alignment between experimental and theoretical variance (Figure 1b) further supports the claim that the network is optimizing for variance reduction.

Discussion and Conclusion

The success of AttEST highlights that for short-text embeddings in e-commerce, simplicity and theoretical alignment often trump architectural complexity. By acting as a variance reducer, attention effectively handles the sparsity and noise inherent in massive commercial datasets. While highly effective for English queries, future work is needed to adapt this "concatenative morphology" approach to languages with different structures, such as Arabic or Hebrew. Additionally, extending this variance-reduction framework to multi-modal settings remains a promising avenue for research.

Summary updated

Abstract

We consider the problem of predicting edges in a graph from node attributes in an e-commerce setting. Specifically, given nodes labelled with search query text, we want to predict links to related queries that share products. Experiments with a range of deep neural architectures show that simple feedforward networks with an attention mechanism perform best for learning embeddings. The simplicity of these models allows us to explain the performance of attention. We propose an analytically tractable model of query generation, AttEST, that views both products and the query text as vectors embedded in a latent space. We prove (and empirically validate) that the point-wise mutual information (PMI) matrix of the AttEST query text embeddings displays a low-rank behavior analogous to that observed in word embeddings. This low-rank property allows us to derive a loss function that maximizes the mutual information between related queries which is used to train an attention network to learn query embeddings. This AttEST network beats traditional memory-based LSTM architectures by over 20% on F-1 score. We justify this out-performance by showing that the weights from the attention mechanism correlate strongly with the weights of the best linear unbiased estimator (BLUE) for the product vectors, and conclude that attention plays an important role in variance reduction.
BibTeX
@misc{dippel2020attention,
  author = {Matthew Dippel and Adam Kiezun and Tanay Mehta and Ravi Sundaram and Srikanth Thirumalai and Akshar Varma},
  title = {{Attention improves concentration when learning node embeddings}},
  year = {2020},
  eprint = {2006.06834},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.06834},
}
Figure