Adversarial Training of Word2Vec for Basket Completion

Ugo Tanielian, Mike Gartrell, Flavian VasilearXiv 2018 · arXiv

Abstract

In recommendation systems, basket completion involves predicting the next item a user is likely to add to their shopping cart. Traditional approaches using Word2Vec rely on static negative sampling, which often fails to provide a sufficiently strong training signal. This paper introduces GAN-Word2Vec, a novel approach that applies Generative Adversarial Networks (GANs) to the task of basket completion. By replacing static negative sampling with a dynamic, adversarial generator that selects challenging negative examples, the proposed method significantly improves the quality of item embeddings and recommendation accuracy. Experimental results on real-world retail datasets demonstrate that GAN-Word2Vec outperforms standard Word2Vec variants in both Precision@1 and Mean Percentile Rank (MPR).

Introduction

The "basket completion" task is central to modern e-commerce recommendation engines. A common solution is to adapt Word2Vec (or Prod2Vec) to learn item embeddings by distinguishing between observed items in a basket and "negative" items sampled from the catalog. However, the standard Negative Sampling (NS) strategy typically draws from a fixed distribution (e.g., unigram frequency), often resulting in "easy" negatives that do not challenge the model. For instance, distinguishing a gallon of milk from a power tool provides little information for refining item representations. This research addresses the challenge of dynamic negative sampling: identifying "hard" negatives that are plausible enough to force the model to learn finer-grained distinctions.

Methodology

The authors propose GAN-Word2Vec, framing the embedding process as a minimax game between two components:

  1. The Discriminator ($D$): The core Word2Vec model, which aims to maximize the probability of "real" items while minimizing the probability of "fake" items provided by the generator.
  2. The Generator ($G$): A model that learns to sample "next-item" candidates that are highly plausible given the current context, aiming to "fool" the discriminator.

Technical Innovations

  • Adversarial Negative Sampling Loss: The discriminator uses an extension of the standard Negative Sampling objective, but with negatives drawn dynamically from $G$.
  • Policy Gradient Training: Since item selection is discrete and non-differentiable, the authors employ reinforcement learning (specifically policy gradients inspired by the MALIGAN architecture) to train the generator.
  • Mixed MLE-MALIGAN Loss: To ensure training stability, a "Mixed Loss" combines the adversarial signal with a standard maximum likelihood estimate (MLE), preventing generator collapse and accelerating convergence.

Experimental Setup

The model was evaluated on two public datasets:

  • Amazon Baby Registries: Registries across 15 categories (e.g., 'feeding', 'toys'), containing up to 15,000 baskets per category.
  • Belgian Retail Supermarket: A larger dataset with 88,163 baskets and a catalog of 16,470 unique items.

Both datasets utilized an 80/20 training/testing split.

Results and Discussion

The performance of GAN-Word2Vec was compared against standard Word2Vec trained with Noise Contrastive Estimation (NCE) and Negative Sampling (NEG).

Performance comparison of GAN-Word2Vec against NCE and NEG baselines on Amazon and Belgian retail datasets.

As shown in the table above, GAN-Word2Vec achieved superior results across both datasets:

  • Amazon Dataset: GAN-Word2Vec reached a Precision@1 of 16.30%, outperforming the NEG baseline (15.40%).
  • Belgian Retail Dataset: GAN-Word2Vec achieved a Precision@1 of 35.82%, compared to 34.35% for the NEG baseline.
  • Mean Percentile Rank (MPR): The adversarial approach showed statistically significant improvements in MPR, indicating that the true missing item was consistently ranked higher.

Key Insights

  • The Value of Hard Negatives: Negatives with high inner-product scores (items the model predicts as related but are absent) provide more informative gradients, leading to more robust representations.
  • Stability via Pre-training: Pre-training both $G$ and $D$ using standard negative sampling was found to be essential for high predictive quality.
  • Generative Performance: Interestingly, the generator itself often provides better performance on the final basket completion task than the discriminator, likely because the task is inherently generative.

Conclusion and Future Work

While GAN-Word2Vec demonstrates the power of adversarial sampling, the authors note that Word2Vec has limited capacity. Future work will explore scaling this framework to more complex architectures like RNNs, LSTMs, or TextCNNs. Additionally, applying this procedure to models designed for sets, such as Determinantal Point Processes (DPPs), could further improve the modeling of basket diversity.

Summary updated

Abstract

In recent years, the Word2Vec model trained with the Negative Sampling loss function has shown state-of-the-art results in a number of machine learning tasks, including language modeling tasks, such as word analogy and word similarity, and in recommendation tasks, through Prod2Vec, an extension that applies to modeling user shopping activity and user preferences. Several methods that aim to improve upon the standard Negative Sampling loss have been proposed. In our paper we pursue more sophisticated Negative Sampling, by leveraging ideas from the field of Generative Adversarial Networks (GANs), and propose Adversarial Negative Sampling. We build upon the recent progress made in stabilizing the training objective of GANs in the discrete data setting, and introduce a new GAN-Word2Vec model.We evaluate our model on the task of basket completion, and show significant improvements in performance over Word2Vec trained using standard loss functions, including Noise Contrastive Estimation and Negative Sampling.
BibTeX
@misc{tanielian2018adversarial,
  author = {Ugo Tanielian and Mike Gartrell and Flavian Vasile},
  title = {{Adversarial Training of Word2Vec for Basket Completion}},
  year = {2018},
  eprint = {1805.08720},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1805.08720},
}
Figure