Attention Mixtures for Time-Aware Sequential Recommendation

Viet-Anh Tran, Guillaume Salha-Galvan, Bruno Sguerra, Romain Hennequin

Abstract

Sequential Recommendation (SR) is critical for personalized user experiences, yet standard Transformer-based models often overlook the nuanced temporal context of interactions. We propose MOJITO (Mixture Of Joint Item and Temporal cOntext), a novel architecture that utilizes Gaussian mixtures of attention to integrate temporal metadata—such as time of day or seasonality—directly into the sequential modeling process. MOJITO addresses the issue of redundant attention heads by sampling attention matrices from a probabilistic model, allowing for dynamic balancing of item sequences and temporal context. Our evaluation across three large-scale datasets shows that MOJITO achieves state-of-the-art performance, particularly in capturing periodic and context-dependent user preferences.

Introduction

Modern recommender systems rely heavily on Sequential Recommendation to predict a user's next interaction based on their history. While architectures like SASRec and BERT4Rec have set benchmarks, they typically treat time as a simple ordinal sequence. In reality, user behavior is deeply tied to context: music choices vary by time of day, and purchasing habits often follow seasonal cycles.

Existing methods that attempt to incorporate time often rely on simple feature concatenation, which fails to capture the complex, non-linear interactions between "what" was done and "when" it occurred. Furthermore, standard multi-head attention often suffers from "redundant heads," where different parts of the model learn identical patterns, wasting computational capacity. MOJITO addresses these challenges by jointly learning from user-item interactions and time-related metadata through a flexible mixture-of-attention framework.

Methodology: The MOJITO Architecture

MOJITO decomposes the recommendation task into two primary representations: short-term and long-term.

Short-Term Representation (Attention Mixtures)

The core innovation of MOJITO is the use of Attention Mixtures instead of standard multi-head attention. Attention matrices are sampled from a Gaussian mixture probabilistic model, enabling the model to automatically learn the relative importance of item sequences versus temporal context for any given prediction. Temporal features (e.g., "Monday" or "10 PM") are mapped into continuous embedding vectors using a translation-invariant time kernel. This approach ensures that the model can dynamically adjust its focus between the "what" (item history) and the "when" (temporal context), preventing it from being locked into a single signal.

Long-Term Representation (Attentive FISM)

To capture stable user preferences, MOJITO employs an Attentive FISM (Factored Item Similarity Model). Rather than a static user vector, the user is represented as a weighted sum of past interactions, allowing the representation to shift based on the target item. This is particularly effective for users with diverse interests, such as a listener who enjoys both Heavy Metal and Classical music.

The MOJITO architecture integrates short-term attention mixtures with long-term attentive FISM components.

Experiments and Results

The model was evaluated on three real-world datasets: MovieLens (1M ratings), Amazon Book (>1M interactions), and LFM-1b (music listening history). MOJITO was compared against eight high-performing baselines, including non-time-aware models (SASRec, BERT4Rec) and time-aware ones (TiSASRec, MEANTIME).

Performance comparison of MOJITO against state-of-the-art baselines across three datasets.

MOJITO achieved the best results on 5 out of 6 primary metrics across the three datasets. On the Amazon Book dataset, it reached a top 83.26% Hit Rate (HR@10) and 59.94% NDCG@10, comfortably beating the next best time-aware model. On the LFM-1b music dataset, it achieved 60.14% NDCG, demonstrating its strength in modeling periodic music listening patterns. While it was a close second in NDCG on MovieLens, it maintained the highest Hit Rate (82.09%).

Conclusion

MOJITO demonstrates that jointly modeling item sequences and temporal context through attention mixtures significantly improves recommendation accuracy. By increasing head diversity and providing a flexible framework for context integration, MOJITO overcomes the limitations of standard Transformers in time-sensitive domains. Future work will explore extending this "Mixture of Context" framework to other metadata types, such as location and device information, to further enhance personalization.

Summary updated

Abstract

Transformers emerged as powerful methods for sequential recommendation. However, existing architectures often overlook the complex dependencies between user preferences and the temporal context. In this short paper, we introduce MOJITO, an improved Transformer sequential recommender system that addresses this limitation. MOJITO leverages Gaussian mixtures of attention-based temporal context and item embedding representations for sequential modeling. Such an approach permits to accurately predict which items should be recommended next to users depending on past actions and the temporal context. We demonstrate the relevance of our approach, by empirically outperforming existing Transformers for sequential recommendation on several real-world datasets.
BibTeX
@inproceedings{tran2023attention,
  author = {Viet-Anh Tran and Guillaume Salha-Galvan and Bruno Sguerra and Romain Hennequin},
  title = {{Attention Mixtures for Time-Aware Sequential Recommendation}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2023},
  doi = {10.1145/3539618.3591951},
}
Figure