Multi-Tower Multi-Interest Recommendation with User Representation Repel

Tianyu Xiong, Xiaohan Yu

TL;DR

This paper introduces the Multi-Tower Multi-Interest (MTMI) recommendation framework, which addresses key limitations of existing multi-interest learning methods. MTMI uses a multi-tower architecture and a novel Inverse Distance Weighting Loss (IDWLoss) to better align training and deployment objectives, leverage item information, and improve industrial adoption. Experimental results show significant improvements over baseline methods.

What Problem Does This Paper Address?

Recommender systems are crucial for navigating information overload, and multi-interest sequential recommendation has emerged as a powerful approach. These systems aim to capture the diverse interests of users by generating multiple user representations. However, current multi-interest learning methods face three major challenges:

  1. Training vs. Deployment Discrepancy: The way user representations are selected during training (e.g., using a target item) differs from how they are used during deployment, leading to suboptimal training.
  2. Limited Item Information Access: Many models struggle to effectively incorporate rich item-side information, treating candidate matching as a simple binary classification problem.
  3. Industrial Adoption Hurdles: The prevalent single-tower architecture of many multi-interest models can be difficult to integrate into existing industrial two-tower recommendation systems.

Methods

The paper proposes the Multi-Tower Multi-Interest (MTMI) framework to tackle these issues. Key components include:

  • Multi-Tower Architecture: Unlike single-tower models, MTMI employs multiple "user towers," each generating a distinct user representation. This allows for a richer capture of diverse user interests. An independent "item tower" is also used to process item information.
  • Inverse Distance Weighting Loss (IDWLoss): This novel loss function is designed to address the training-deployment gap. It uses cosine similarity and distance to pull user representations towards the target item representation while simultaneously pushing them away from each other (user representation repel). This encourages distinct interest representations and better aligns with how these representations will be used for candidate matching.
  • User Representation Repel: This mechanism, integrated within the IDWLoss, encourages the different user representations generated by the multiple towers to be distinct from each other. This helps to better capture the user's varied interests.
  • Adaptability to Two-Tower Systems: MTMI is designed to be compatible with industry-standard two-tower candidate matching architectures, facilitating easier adoption.

Datasets

The authors evaluated their framework on three large-scale, real-world industrial datasets:

  • MovieLens 100k: A well-known dataset for movie recommendations.
  • Retail Rocket: An e-commerce dataset focusing on view events.
  • Amazon Books: A subset of the Amazon product view dataset, specifically focusing on books.
  • Gowalla: A location-based social networking dataset.

The paper notes that dataset preprocessing was done in alignment with prior research [4, 5, 9], including filtering users/items with low interaction counts and setting a maximum sequence length.

Key Insights and Ideas

  • The "Repel" is Key: The core innovation lies in not just capturing multiple interests but actively encouraging these interests to be distinct. This "repulsion" mechanism seems crucial for preventing representations from collapsing and for truly reflecting diverse user preferences.
  • Bridging the Training-Deployment Chasm: The IDWLoss is a clever way to make the training objective more closely mimic the deployment scenario, where multiple user representations are used independently to find similar items.
  • Item Tower for Richer Features: Having a dedicated item tower allows the model to leverage item features more effectively, moving beyond simple item IDs and addressing the limitation of not fully utilizing item-side information.
  • Practicality Matters: The framework's design to integrate with existing two-tower systems is a significant practical advantage for industrial applications.

Key Results

  • MTMI Enhances Two-Tower Models: When applied to existing two-tower models (YouTube DNN, GRU4Rec, Que2Search), the MTMI-enhanced versions showed significant performance improvements across various metrics (Recall, Hit Rate, NDCG) on the Books, Gowalla, and Retail Rocket datasets.
  • MTMI Outperforms Basic MIL: MTMI-GRU also outperformed basic multi-interest learning (MIL) models like MIND and ComiRec.
  • IDWLoss Effectiveness: Ablation studies showed that the IDWLoss significantly outperformed simpler loss functions like triplet loss and label-aware attention, especially in preventing performance degradation.
  • User Representation Repel Benefits: Turning on the User Representation Repel mechanism also led to demonstrably better performance.
  • Hyperparameter Sensitivity: The paper conducted extensive hyperparameter studies, finding optimal values for parameters like the number of towers (K), distance scaling coefficient (α), and divergence parameter (β).

Limitations and Future Work

  • Not Yet State-of-the-Art: While MTMI shows significant improvements, the authors acknowledge that it does not yet surpass the absolute state-of-the-art in multi-interest learning.
  • Future Directions: The authors suggest several avenues for future work:
  • Refining hard negative mining strategies.
  • Developing more sophisticated and balanced loss functions.
  • Incorporating more effective inter-tower communication mechanisms.
  • Exploring enhancements to the user representation repel mechanism.

Summary updated

Abstract

In the era of information overload, the value of recommender systems has been profoundly recognized in academia and industry alike. Multi-interest sequential recommendation, in particular, is a subfield that has been receiving increasing attention in recent years. By generating multiple-user representations, multi-interest learning models demonstrate superior expressiveness than single-user representation models, both theoretically and empirically. Despite major advancements in the field, three major issues continue to plague the performance and adoptability of multi-interest learning methods, the difference between training and deployment objectives, the inability to access item information, and the difficulty of industrial adoption due to its single-tower architecture. We address these challenges by proposing a novel multi-tower multi-interest framework with user representation repel. Experimental results across multiple large-scale industrial datasets proved the effectiveness and generalizability of our proposed framework.
BibTeX
@misc{xiong2024multi,
  author = {Tianyu Xiong and Xiaohan Yu},
  title = {{Multi-Tower Multi-Interest Recommendation with User Representation Repel}},
  year = {2024},
  eprint = {2403.05122},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.05122},
}
Figure