Item-based Collaborative Filtering with BERT

Tian Wang, Yuyangzi FuACL 2020

TL;DR

This paper introduces a novel item-based collaborative filtering approach that leverages BERT's language understanding capabilities to represent items by their titles. This method effectively addresses the cold-start problem and improves recommendation quality by capturing deeper item semantics beyond simple identifiers.

What Problem Does This Paper Address?

Recommender systems are crucial for e-commerce, helping users discover products. Traditional collaborative filtering methods, both user-based and item-based, struggle with data sparsity and the "cold-start" problem, where new items or users have little to no interaction data. Content-based methods, while better with sparsity, can lead to recommendations that are too similar and lack novelty. The dynamic nature of e-commerce, with constantly changing inventory, exacerbates these challenges, requiring frequent retraining and redeployment of systems. This paper aims to overcome these limitations by using BERT to understand item titles and model relevance, thereby improving cold-start performance and recommendation quality.

Methods

The core of this paper is the adaptation of BERT, a powerful pre-trained language model, for item-based collaborative filtering. Instead of relying on unique item IDs, the authors represent each item using its title tokens. This approach allows the model to leverage BERT's transformer architecture and attention mechanisms to understand semantic relationships between item titles.

The authors fine-tune a pre-trained BERT model on two tasks adapted for the e-commerce context:

  • Next Purchase Prediction: This task aims to predict the next item a user will purchase given a seed item they just bought. Item titles are treated as "sentences" A and B, and the model learns to score the relevance between them.
  • Masked Language Model (MLM): To better adapt BERT to the domain-specific language of item titles, the model is further fine-tuned using the MLM task. This involves masking tokens in item titles and predicting them, helping the model understand the nuances of e-commerce vocabulary.

The final model is optimized using a joint loss function combining both MLM and Next Purchase Prediction. For comparison, a Bi-LSTM baseline model is also implemented, which uses title token embeddings, a bidirectional LSTM, and an MLP.

Datasets

The authors train and evaluate their models on a large-scale e-commerce dataset consisting of 8,001,577 co-purchased item pairs. 33% of these pairs were co-purchased within the same user session, while the rest were randomly sampled as negative examples. The dataset exhibits extreme sparsity, with 99.9999% of the item-item interaction matrix being empty. A validation set of 250,799 pairs was used for early stopping. For testing, a cold-start scenario was simulated by sampling 10,000 co-purchased item pairs where the seed item was not present in the training set.

Key Insights and Ideas

  • Item Titles as Rich Semantic Representations: Representing items by their titles, rather than just IDs, allows BERT to capture nuanced semantic similarities that traditional methods miss. This is crucial for understanding user preferences beyond simple co-occurrence.
  • BERT's Adaptability to Domain-Specific Language: Fine-tuning BERT on e-commerce item titles (using MLM) is essential. The language used in product titles differs significantly from general natural language, and this adaptation is key to unlocking BERT's full potential for recommendations.
  • Addressing Cold-Start Effectively: By leveraging title similarity, the proposed BERT-based approach can recommend new items even when they have no interaction history, a significant improvement over traditional methods.
  • Beyond Simple Similarity: The model appears to automatically learn complex selection criteria and can decide whether to focus on a specific category or offer a broader range of items, demonstrating a sophisticated understanding of user interests.

Key Results

The proposed BERT model significantly outperforms the Bi-LSTM baseline across various metrics:

  • Next Purchase Prediction Fine-tuning: The BERT model (without MLM fine-tuning) achieved improvements of 310.9% (Precision@1), 96.6% (Precision@10), 93.9% (Recall@10), and 150.3% (NDCG@10) over the baseline.
  • Joint Fine-tuning (MLM + Next Purchase Prediction): Further fine-tuning with MLM led to even greater improvements, adding another 111.0% (Precision@1), 38.6% (Precision@10), 38.3% (Recall@10), and 64.0% (NDCG@10).

Qualitative analysis showed that for a "Marvel Spiderman T-shirt" seed item, recommendations included other Marvel-themed apparel and accessories. For a "Microsoft Surface Pro 4" seed item, recommendations spanned a diverse range of categories including electronics, accessories, and digital services, demonstrating the model's ability to infer relevant criteria.

Limitations and Future Work

The paper acknowledges that traditional item-to-item collaborative filtering models and approaches relying solely on unique item identifiers were not considered in the evaluation due to the complete cold-start setting. This is a practical limitation for real-world scenarios where such models might still be used for items with existing data. The authors suggest that their approach is a practical experiment setting for large-scale e-commerce platforms where new items are constantly introduced. Future work could explore incorporating other item attributes beyond titles and further investigating the model's ability to balance novelty and relevance.

Summary updated

BibTeX
@inproceedings{wang2020item,
  author = {Tian Wang and Yuyangzi Fu},
  title = {{Item-based Collaborative Filtering with BERT}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure