Every Preference Changes Differently: Neural Multi-Interest Preference Model with Temporal Dynamics for Recommendation

Hui Shi, Yupeng Gu, Yitong Zhou, Bo Zhao, Sicun Gao, Jishen ZhaoarXiv 2022 · OpenReview

Abstract

Traditional recommendation systems often represent users as single vectors, failing to capture the diversity and varying temporal dynamics of individual interests. We present the Multi-Interest Preference (MIP) model, which utilizes clustering-enhanced self-attention to generate multiple interest embeddings. By incorporating a learned weighting mechanism that accounts for temporal patterns, MIP effectively captures how different interests evolve over time. Evaluations on large-scale public and industrial datasets demonstrate that MIP significantly outperforms state-of-the-art multi-interest models in retrieval performance.

Introduction

Modern industrial recommendation systems typically employ a "retrieve-then-rank" architecture. In the initial retrieval phase, the system must efficiently filter billions of items down to a manageable set of candidates. Historically, this has been achieved by mapping users to a single embedding vector. However, this approach is fundamentally limited because human interests are multifaceted. A single "average" embedding for a user interested in both heavy metal and knitting may result in a representation that resides in a semantic "no-man's-land," leading to poor retrieval quality.

While existing multi-interest models attempt to address this by generating multiple vectors, they often suffer from two critical weaknesses: uniform weighting and static temporal decay. They typically treat all interests as equally important and assume that every interest fades at the same rate. In reality, a user's interest in a specific hobby may last years, while interest in a one-time purchase might be transient.

Methodology

The Multi-Interest Preference (MIP) model addresses these limitations through a three-part architecture designed to capture distinct interests and their unique temporal signatures.

Clustering-Enhanced Multi-Head Attention

Instead of relying on a global query vector—which often biases results toward globally popular items—MIP utilizes self-attention. The model projects a user's interaction history into multiple interest spaces. A clustering algorithm (such as Ward or K-Means) then groups these projections into $\Lambda$ distinct interest embeddings. This allows the "query" for retrieval to be derived directly from the user's specific behavioral history.

Learned Interest Weights and Temporal Dynamics

A core innovation of MIP is the use of a Feed-Forward Network (FFN) to automatically learn weights for each interest cluster. These weights are not static; they are functions of the interest embedding and the specific timestamps of items within that cluster. This enables the model to learn that some categories are "sticky" while others are "transient" for a given user.

The MIP architecture consists of three main stages: Multi-interest User Embedding, Cluster Weighting, and User-Item Interaction.

Training and Inference

To ensure stable convergence, the authors employ a two-stage training strategy. Initially, the model is trained with all cluster weights fixed to 1 to allow distinct interest clusters to form. Subsequently, the weights are unfrozen, and the full system is optimized. Furthermore, the model offers flexible inference, allowing the number of clusters to be adjusted dynamically to balance computational cost and performance without retraining.

Experimental Evaluation

The MIP model was evaluated against state-of-the-art baselines, including GRU4Rec, MIND, ComiRec, and PinnerSage, across four large-scale datasets: Amazon-book (51M interactions), Taobao (100M interactions), MovieLens-20M, and an industrial Pinterest dataset (38M interactions).

Key Results

MIP consistently outperformed all baselines. On the Amazon dataset, MIP achieved a Recall@50 of 0.789, a significant improvement over the next best baseline, TiSASRec (0.667). On the industrial Pinterest dataset, MIP improved Recall@20 from 0.296 (PinnerSage) to 0.353 and drastically reduced the Negative Log-Likelihood (NLL) from 1.033 to 0.377.

Comparison of MIP's learned weighting scheme against heuristic exponential decay and equal weighting across public datasets.

As shown in the evaluation metrics, the learned weighting scheme in MIP (MIP) significantly outperforms both uniform weighting (equal weight) and standard heuristic decay (exp decay). This confirms that learning personalized temporal dynamics is superior to applying a one-size-fits-all decay rate to all user interests.

Conclusion

The MIP model represents a significant step forward in personalized retrieval by acknowledging that every preference changes differently. By combining self-attention with learned, time-aware cluster weights, MIP provides a more nuanced and effective representation of multifaceted user behavior. Future work may explore addressing the cold-start problem for new users and further optimizing the computational overhead of the clustering step in real-time production environments.

Summary updated

BibTeX
@misc{shi2022every,
  author = {Hui Shi and Yupeng Gu and Yitong Zhou and Bo Zhao and Sicun Gao and Jishen Zhao},
  title = {{Every Preference Changes Differently: Neural Multi-Interest Preference Model with Temporal Dynamics for Recommendation}},
  year = {2022},
  url = {https://openreview.net/forum?id=aLq_K3wn2d},
}
Figure